Map data lump order

WAD / Map_data_lumps">

WAD / Map_data_lumps said:

The first lump gives the internal name of the map. [...] In order to work properly, the following lumps must follow immediately after the level name:
(THINGS, LINEDEFS, SIDEDEFS, VERTEXES, SEGS, SSECTORS, NODES, SECTORS, REJECT, BLOCKMAP, BEHAVIOR)

The wiki doesn't say whether they have to follow in this exact order. Can for example THINGS and LINEDEFS lumps switch places? Are source ports expected to load such maps properly? Are map editors, nodebuilders and similar utilities allowed to output maps with lumps in such a different order? Should automatic wad editing utilities take into account the possibility that map data lumps after a map marker in the input wad could be in any order?

The lump order is absolutely important, and Slade3 is built with exact specifications (it wouldn't be a very good Doom lump management program if it wasn't). The order is mostly in stone because Doom needs to find lumps relative to the map marker, and cannot do a normal global search. As the lump data needs to be loaded in a specific order (sectors before sidedefs before lindefs, for example), it's much easier to assume the data is where it always is.

Source ports change this with a setup check to find each lump, but of note this appears to just be ZDoom based and Eternity (only partially). As such it's probably a bad idea to assume it can be changed. Note that they are the only two ports that support UDMF, which has no order as a matter of spec, as long as it ends with ENDMAP.

Thanks. It cleared up the issue with source ports that actually load the game. But what about other utilities that work with wads, such as map editors and nodebuilders? I'm asking because I'm programming a node builder from scratch and I need to decide whether it should skip maps with a non-standard lump order or whether it should process them and rearrange the lumps into the correct order.

Why would you even be running into maps with a nonstandard lump order?

That's what I'm asking for, if such a situation may happen or not. For example if some map editor produces such maps while they don't have nodes built yet.

scifista42 said:

That's what I'm asking for, if such a situation may happen or not. For example if some map editor produces such maps while they don't have nodes built yet.

The standard usual behavior is to generate empty lumps. So you'll see NODES, SEGS, SSECTORS, etc, all of zero length. The BSP builder should then fill in those entries with size and offset after adding its data. The VERTEXES lump must also be modified with any additional vertices created during seg splitting.

Quasar said:

The standard usual behavior is to generate empty lumps.

I'm pretty sure I've come across wads that, when opened in SLADE3, turned out to contain only the map header and THINGS, LINEDEFS, SIDEDEFS and SECTORS lumps.

That might happen if you don't run a nodebuilder at all.

May I ask a similar question?

edit: OK, thanks Gez.

So if map editors (before running a nodebuilder) sometimes produce maps containing zero-length NODES, SEGS and SSECTORS, and sometimes don't produce these lumps at all, those are already 2 standards that a nodebuilder needs to check for in order to properly process the map when building nodes for it. Are these standards arbitrary, meaning that a map without nodes built yet could contain lumps in any order and be either containing or missing any lump except for the 5 essential ones?

bzzrak said:

May I ask a similar question?

Spoiler

Do you have to include all of the map lumps?
E.g. Can I, for example, fix a texture misalignment in a WAD by only providing a SIDEDEFS (or whatever is required) lump? (with a map marker, if necessary.)

You have to include all the map lumps. If you load a map with just a SIDEDEFS lump, the engine will not think "no matter, I'll just look if there's another level of the same name and load its stuff", it'll abort/crash right there.

The lump ordering is more clearly visible here

https://github.com/chocolate-doom/chocolate-doom/blob/bd7a22c321d9fba5922f4c5c62b403376d88e039/src/doom/doomdata.h#L40

WadC generates the following lump order

MAP01
THINGS
LINEDEFS
SIDEDEFS
VERTEXES
SECTORS
BEHAVIOR (if hexen format)
WADCSRC

It doesn't put in placeholder NODES/etc lumps.

glbsp inserts them in the right place, and appends GL lumps

MAP01
THINGS
LINEDEFS
SIDEDEFS
VERTEXES
SEGS
SSECTORS
NODES
SECTORS
REJECT
BLOCKMAP
BEHAVIOR
GL_MAP01
GL_VERT
GL_SEGS
GL_SSECT
GL_NODES
GL_PVS
WADCSRC

scifista42 said:

So if map editors (before running a nodebuilder) sometimes produce maps containing zero-length NODES, SEGS and SSECTORS, and sometimes don't produce these lumps at all, those are already 2 standards that a nodebuilder needs to check for in order to properly process the map when building nodes for it. Are these standards arbitrary, meaning that a map without nodes built yet could contain lumps in any order and be either containing or missing any lump except for the 5 essential ones?

I am not aware of any editor that spits out the lumps in a different order, but the de-facto standard is probably whatever the popular node builders of the time accepted. So I'd read the BSP and ZENNODE source code.

OK, so I take it that there's no official standard for the lump order of maps without nodes built yet, and that map editors and node builders are compatible with each other merely thanks to unorganized adjusting to each other's input/output format specifics, modeled after those of some previously existing program's of their kind. Which means that programming a new node builder's input logic isn't about conforming to some universal standard, but just about conforming to the output of particular map editors that are being used nowadays, which would be most safely done by copying the exact logic of a particular node builder that is being used nowadays. Correct me if I'm wrong. Assuming I'm right, I have one more question: If I program the input logic to accept and process maps with lumps in any order (in attempt to avoid incompatibility with any past or future map editors, without knowing anything about those editors), would anything undesired happen, like accepting and processing lumps that aren't really a map and therefore shouldn't have been processed? At this point, I'm inclined to believe that accepting and processing maps with lumps in any order is the ideal way how an universal node builder should work. But please correct me if I'm missing something.

There is a standard order. The only thing is whether some lumps are skipped or not, but it doesn't change the ordering of the rest.

THINGS
LINEDEFS
SIDEDEFS
VERTEXES
(SEGS)
(SSECTORS)
(NODES)
SECTORS
(REJECT)
(BLOCKMAP)
(BEHAVIOR)

Thanks. So, allowing certain lumps to be missing (as long as they're not essential to build nodes), while requiring their strict relative order (and aborting upon encountering a lump that breaks it), should be an universally acceptable node builder behavior? If so, I'll go along with it.

Vanilla IDTech1 engines absolutely require an exact ordering of those lumps. In fact, the map marker is the only lump name that vanilla engines look for by name. They expect the very next lump to hold THING information, then LINEDEF info, etc.

Some source ports can handle the omission of some of those lumps, and the addition of other custom lumps. Those ports have specialized code for checking for lump names following the map marker. But, ,even those ports expect some of the lumps to be in a rigid order.

To scientifically answer your question exactly, whatever tool you are building should read its lumps the exact same way the source ports that you're supporting do.

However, you could really get away with reading in the lumps Gez listed, that are not in parenthesis. Your node builder will rebuild the lumps in parenthesis. A GL node builder will also rebuild the additional "GL_" lumps in Jon's post.

When writing the lumps back into the wad, you'd want to delete the entries in parenthesis, replacing them with newly generated lumps. Same goes for the "GL_" lumps. (If your node builder does not generate GL nodes, you may prompt the user on whether or not to delete the GL nodes. If the user has changed the map, there's no reason not to delete the GL nodes, cause they're no longer accurate, most likely.

The resultant wad's lumps MUST BE in the proper order, as shown in Gez's and Jon's posts. If lumps are omitted, you must use a source port that tolerates that.

<soapbox>
Oh, and for the sake of your players, if you are creating a node builder, please include a REJECT builder. The only downsides are that the map build takes a bit longer, and the wad is a little bit bigger. The upside is that the gameplay is almost always faster, and, at worst, is exactly as fast.

</soapbox>
EDIT: If the map has a BEHAVIOR lump, you should load that in too, as it could contain ACS scripts used in Hexen for custom scripting.