Flat replacement bug

[Jon]

Just noticed that eternity doesn't handle swapped in flats in some circumstances



this pic from TTPDEATH.WAD map02. You need to get ttp.zip from idgames. I then modified it by deleting all the sprite lumps and the S_END and F_END markers. (I was doing that to get it running in chocolate, the sprites caused problems)

Here's the same modified WAD in doom2.exe v1.9



the F_END marker is not necessary for the flat substitution to work in Vanillla

(imgur gallery for the two shots http://imgur.com/a/G9aMo)

[Graf Zahl]

How do other ports handle that? I think this is a direct consequence of Boom's namespaces, not something in Eternity.

[Jon]

Chocolate-* work as expected (but doesn't inherit the boom stuff); prboom+ 2.5.1.3 segfaults on startup for me but I'll try building it from source.

[printz]

I guess I should modify the F_START/F_END lookup to be compatible with vanilla? OK. I wouldn't deny TeamTNT changed it.

[Jon]

just got prboom-plus built and it behaves as Eternity!

[VGA]

Where is the modified wad you used for testing?

[Graf Zahl]

VGA said:

Where is the modified wad you used for testing?

Good question. I'd like to run it through ZDoom as well.

[Gez]

Jon said:

I then modified it by deleting all the sprite lumps and the S_END and F_END markers. (I was doing that to get it running in chocolate, the sprites caused problems)

If the sprites caused problems, why did you also delete F_END?

[Jon]

Gez said:

If the sprites caused problems, why did you also delete F_END?

I just thought it was superfluous. I originally just deleted it because I wanted to get the thing running, I wasn't trying to explore this bug - I hadn't discovered it yet :)

The precise WAD I can't distribute because TTP doesn't permit modification or redistribution. This is TTP: https://www.doomworld.com/idgames/levels/doom2/s-u/ttp103 and it's the PWAD TTPDEATH.WAD specifically (and those screenshots are from MAP02)

That's why I wrote the recipe down instead, it should only take a few minutes to delete those entries from the PWAD in SLADE or whatever. I (or someone else) could make a separate, unrelated test map, it just requires a single flat replacement and no F_END as far as I can see.

[printz]

1. Vanilla Doom can't load this as is. Instead it requires the help of a tool called "nwt" (I know what it is) to "merge" it.
2. PrBoom+ acts like in the Eternity screenshot. And Eternity acts like PrBoom+ here.
3. But ZDoom loads the flats (but not the sprites) properly. This unfortunately keeps the issue active, I can't close it yet. I wonder how to fix it then, without hacking the engine apart.

Apparently vanilla Doom uses the last F_START it can find in the whole wad collection (which is only the one from the IWAD) and the last F_END (which is not the one from the IWAD, but the one from the PWAD). Need to think how to support that without breaking apart the nicer Boom namespace scheme.

[Quasar]

Never mind the flats even, what in the name of god is going on at the top edge of that black and gray texture???

[Graf Zahl]

2 hours ago, printz said:

1. Vanilla Doom can't load this as is. Instead it requires the help of a tool called "nwt" (I know what it is) to "merge" it.
2. PrBoom+ acts like in the Eternity screenshot. And Eternity acts like PrBoom+ here.
3. But ZDoom loads the flats (but not the sprites) properly. This unfortunately keeps the issue active, I can't close it yet. I wonder how to fix it then, without hacking the engine apart.

Apparently vanilla Doom uses the last F_START it can find in the whole wad collection (which is only the one from the IWAD) and the last F_END (which is not the one from the IWAD, but the one from the PWAD). Need to think how to support that without breaking apart the nicer Boom namespace scheme.

What I did in ZDoom was relatively simple after finding out that all the hacks trying to make this work with namespaces showed other issues:

I flag every lump in an F_START-less WAD before F_END that is exactly 4096 bytes and then, when creating the flat textures, not only create ones for the actual lumps in the ns_flats namespace but also for ones that got flagged this way. It's still not 100% foolproof because it'd skip anything that's not properly sized but ever since I added this workaround I got no more reports.

 

The only caveat to observe here is that ZDoom does not sort the lump directory by namespaces to ensure that everything remains in proper order (with sorting that scheme would not work without forcing ns_flats onto these lumps which has its own share of problems.)

 

For sprites this cannot be done, because a) the engine performs much stricter checks which must lumps would fail and b) it would force any other lump that can be identified as a graphics file into the sprites namespace.

 

[Gez]

Chocolate Doom has a -nwt parameter that you can use to load files that need nwt.

[Altazimuth]

3 hours ago, Quasar said:

Never mind the flats even, what in the name of god is going on at the top edge of that black and gray texture???

That's intended, see pic below of map open in GZDB, showing what is on the left of the screenshot in OP.

[printz]

This won't be fixed, because the wad requires doctoring in order to run in vanilla Doom in the first place. Also, PrBoom+ doesn't have a solution either, and GZDoom's is rather hacky and not 100% successful.

[kb1]

Removing S_END and F_END from your PWAD creates a situation where the port encounters the IWAD's S_END and F_END markers before it encounters S_START and F_START, which, in my opinion, is an invalid construct.

 

Boom ports find flats using a chained hash, so it can find them anywhere, but sprites are referenced relative to the position of the last-encountered S_START. That means that in your setup, the IWAD sprites end up with a negative offset which is not good in general.

 

Many ports have devised schemes to try to handle messed up situations like this. Boom does a second search to resolve textures with patch names that match sprite names. But your setup forces the port to run through all the lumps trying to verify sprites, pre-load textures, etc. S_END and F_END from your PWAD tells the engine where in the combined WAD directory to stop when parsing through them.

 

My point is that, yeah, maybe ports should stop expecting any assistance from the PWAD author, and be able to exhaustively search from middle to bottom, and then roll over back to the top and down through the combined WAD structure to resolve flat and sprite names, assuming that every lump is potentially a flat and a sprite, but, in my opinion, ill-formed PWADs are ill-formed, and behavior should be considered undefined in these situations.

 

This means that, if you don't form your PWAD properly, you cannot rely on how the port behaves. Leaving out the markers is, at least, detrimental to performance, and, at worst, prevents the port from being able to determine which lumps are flats, sprites, or images. The only way the port can know that FLAT1234 is a flat, is that, when loading a map, a sector has asked for "FLAT1234" to be a floor flat. Only now can the port find FLAT1234 and try to load it as a flat.

 

Ports like ZDoom allow flats to be placed on walls too, so it's important for the port to know ahead of time that FLAT1234 is a flat lump. Removing S_END and F_END from your PWAD makes this task pretty difficult.

 

See Graf's post: Extreme hacks are required to attempt to handle horribly invalid PWADs, and it can't fix everything.

 

Build the PWAD properly: Put the markers back - that's the fix.

 

Edited April 27, 2018 by kb1