32-Bit Map Format Proposal

I have completed a draft of an extendable 32-bit map format specification which would address issues such as tight data type limitations, cross-port compatibility, and the storage of implementation-dependent data in DOOM WADs.

The draft specification is submitted here for the review of peers in the source port and editor authorship domains. Comments will be greatly appreciated.

See the draft here:
http://doomworld.com/eternity/engine/wide_map_format_draft.txt

Have you contacted other port authors about this?

I will take a closer look at this spec when I can. First impressions: having a standard header on each lump is a good thing. Wasting 20 bytes per record for unused args (Doom maps) is a bad thing. I also think using text strings for some parts would give greater flexibility.

BTW, there was a discussion on a NewDoom forum about creating a standard for textures (use of PNGs, etc). The following spec was borne out of that:

http://glbsp.sourceforge.net/JointTextureSpec.txt

There wasn't consensus on all of it, most controversial was the handling of flats in F_START (whether to scale to not, whether to allow PNG or not). The text-based format is not finished yet.

Your point on args is duly noted, and I myself had concerns about the size increase that these alone add. A suggested solution to the problem would be appreciated though ;)

This is a copy of my post in the same thread in the ZDoom forum so that my suggestions can be discussed here as well:


Why not do a text based format? That would make things much easier, eliminate most of the need for version control and also makes many of the awkward multi-format handling unnecessary. And if a database for property was created it could be ensured that if a feature is supported it is done the same way everywhere.

But if you'd like to keep it binary I have a few suggestions:

Most importantly, remove and mention of 'implementation specific' from the draft. Every known feature that should be supported should have a unique representation in the data. The moment you leave the specific use of flags and other properties to each port's developer we will quickly get the same mess as we have now. This should be avoided in the planning stages already.

For example:

- The thing flags word should clearly assign a specific function to each bit. If 32 is not enough extend it to 64 bits. If you want you can reserve some flags for port-specific use.

- The specials for things and lines should clearly differentiate between Doom-format and Hexen-format. If the records themselves are the same size for both I recommend against using the presence of BEHAVIOR as the deciding factor. And Hexen-type specials should be standardized so that every implementation must adhere to the same name and function for each number. For Doom-format it is too late to do that but for Hexen it is not.
- For Hexen format I recommend a slightly different layout of linedefs. Currently the trigger type of a Hexen line special is stored as a 5-bit bitfield in the flags word. This not only takes away 5 bits that could be used differently, it also allows only one type of trigger. Since there is hardly any need to use all of the 32 bits to define the special itself (16 bit are more than enough,) how about using the special word as follows:

Bit 31: always 1 to distinguish from Doom format
Bit 30: mark as repeatable
Bits 16-29: Trigger type bit field. ZDoom currently allows 8 trigger types so this leaves sufficient room for expansion.
Bits 0-15: The special itself

For Doom format it just would store the special's value as always. (But since we are defining a new format there is nothing that would prevent us from reassigning colliding uses of certain numbers and make sure this doesn't happen again by giving each port a defined range of numbers it can use for its own purpose.)

- How about officially drafting an extension mechanism that doesn't require version maintenance for the different record types? You can be absolutely certain that this will be used in a manner that will become impossible to support by ports that are interested in being compatible.

To support extended but rarely used features you could add them after the record they belong to by inserting something like:

identifier value

where 'identifier' is a 4 character token that names the property (and should be registered by the commitee you suggest) to be defined and 'value' is the value being assigned (the format depends on the property.) These are easy to skip in the loading process and avoid having to define extended versions of the various records for each new feature.
With such a means to define variable feature sets the format would become much better IMO. Different versions of the base records themselves should be avoided unless there really is some need to. It also has the advantage that features that are normally used only in a small set of sectors don't add baggage to the larger number of sectors where they are not needed. The list should be terminated with a 0-token.



One thing that can be omitted IMO is the BLOCKMAP lump. Even for the largest maps I know the internal blockmap builders need only fractions of a second to build it so there really is no need to waste space for it. AFAIK the only port which doesn't have a BLOCKMAP builder included is Legacy so just for that I see no need to implement it.

Regarding nodes, there's some room to save space. Most of the properties of a seg can be calculated at load time. There is no need to store the angle and the offset. Both values can be quickly retrieved from the linedef they belong to. Reading these from the lump only makes them susceptible to bugs in the node builders. (ZDBSP and WARM definitely have problems with the offset value.) Even the direction is easily calculated by checking the distance to the linedef's vertices so you can cut down a seg's size from 24 bytes to 12 bytes without sacrificing any information (and as we all know, segs can become by far the largest resource in a WAD.)

My 2pence worth:

I would prefer to see that script references (the TID in Things for example) should actually be a string. This would much more informative for the people who build maps. Perhaps a string lump ("MAPSTR"?) should contain all the variable length strings including script references so that thing entries themselves would be easier to read. Therefore the TID entry would actually point to an entry in the string lump. I grant you that numeric IDs are efficent for comparision purposes, but it can be boiled down to that internally by the given source port or editor.

I would suggest that extended information be handled in an additional lump for each type (one for things, lines, sectors etc...).

Text format maps? Nice idea, be interested how big one of these would get with some of the bigger maps we have seen.

DarkknightXC said:

My 2pence worth:

I would prefer to see that script references (the TID in Things for example) should actually be a string. This would much more informative for the people who build maps. Perhaps a string lump ("MAPSTR"?) should contain all the variable length strings including script references so that thing entries themselves would be easier to read. Therefore the TID entry would actually point to an entry in the string lump. I grant you that numeric IDs are efficent for comparision purposes, but it can be boiled down to that internally by the given source port or editor.

That would be ok if they could be automatically referenced by a string inside the game. But with ACS that is not the case so doing it like you suggest would only introduce a level of indirection that may cause needless errors. Unfortunately this is not easily changed.

I would suggest that extended information be handled in an additional lump for each type (one for things, lines, sectors etc...).

What a mess. That would only fragment the data and bloat the thing even more. There has to be a better solution. The current setup has the great advantage that lump management tools that don't care about the level's contents can still handle it. If there were more lump that advantage would be gone.

Text format maps? Nice idea, be interested how big one of these would get with some of the bigger maps we have seen.

With ZLIB compression not that much larger than compressed binary data, depending on the format. But the added flexibility would far outweigh the needed work to decompress it when loaded. If it's good for Doom3 why not for the original Doom?

Graf Zahl said:

That would be ok if they could be automatically referenced by a string inside the game. But with ACS that is not the case so doing it like you suggest would only introduce a level of indirection that may cause needless errors. Unfortunately this is not easily changed.

Ultimately, that is an implementation issue. ACS is one of many which would use such data. I think its makes sense for references to be strings for the reasons stated.

Graf Zahl said:

What a mess. That would only fragment the data and bloat the thing even more. There has to be a better solution. The current setup has the great advantage that lump management tools that don't care about the level's contents can still handle it. If there were more lump that advantage would be gone.

Don't beat around the bush, do you? I'm making a suggestion, not writing gospel. No need to be quite some combative in your language.

I think that record sizes should be fixed, so data maybe read as a block. Its more efficent and easier to navigate. I applaud the acceptance of implementation specific data, but I would have it positioned after the block of standard implementation data. The suggestion made was a potential solution. How would you tackle this problem?

Graf Zahl said:

With ZLIB compression not that much larger than compressed binary data, depending on the format. But the added flexibility would far outweigh the needed work to decompress it when loaded. If it's good for Doom3 why not for the original Doom?

Fair point. I certainly not against text format maps - its just another parser to write that is all.

DarkknightXC said:

Ultimately, that is an implementation issue. ACS is one of many which would use such data. I think its makes sense for references to be strings for the reasons stated.

I thought a little about this how one could translate named tags into the current system.
It would require implementing a TagForName function for those scripting languages which only can handle numeric tags. Then you can have your named items and they can be used directly in the level.
Numbers should still translate directly though.

I think that record sizes should be fixed, so data maybe read as a block. Its more efficent and easier to navigate.

But only as long as you don't have to handle additional properties. Considering that the format being proposed is strictly 4-byte aligned I don't see a problem. You can access it all with a DWORD pointer without the need of awkward type casts or alignment issues as is the case with the current map format. You can load the whole data into memory and step through it DWORD by DWORD.

I applaud the acceptance of implementation specific data, but I would have it positioned after the block of standard implementation data. The suggestion made was a potential solution. How would you tackle this problem?

I already made the suggestion of adding named properties after each record (names being 4 characters long, i.e. 1 DWORD) Sure, that would mean 4 empty bytes for each record as a terminator of the list but it would keep all the data grouped together and any source port which is unable to handle it can either ignore it, print a warning or abort. But it would allow adding all the neat stuff directly into the map data.

And as I said, don't underestimate the importance of support from existing WAD tools. By adding more lumps you will break them. Not all of these tools are actively maintained so they couldn't handle a map format that uses a different lump layout

Fair point. I certainly not against text format maps - its just another parser to write that is all.

Whether you parse a text file or read a stream of binary data that has (at least partially) to be interpreted isn't such a big difference IMO. I don't think that writing the parser would take that much longer. (And on the other hand would make adding string properties much easier than handling some indices into a string table.)

Yeah I think a text file based format is better than a binary one since even though it is a higher overhead it's easier for humans to debug. Also plenty of tools already exist for text that would require the writing of specific utilities for binary formats - ever wished you could grep or diff or patch wad data? There's plenty of free text parsers around too (has anybody made an xml format for wads? or do you hate xml on principle? :) )

I don't know about missing out the blockmap since, although the port building it itself is feasible, slight differences in algorithms between engines or even between different versions of the same engine could play merry havoc with demo compatibility. I mean you can't rebuild the nodes on a map and not screw up demos so I assume blockmaps are the same in that respect. I think it's important the wad contains pre-built data just for that reason, even if it does seem like extra bloat.

RjY said:

I don't know about missing out the blockmap since, although the port building it itself is feasible, slight differences in algorithms between engines or even between different versions of the same engine could play merry havoc with demo compatibility.

Demo compatibility between source ports? I really don't think that's really an issue because no source port interested in such a map format is demo compatible with any other. And if using the same port the blockmap will be built the same.

I mean you can't rebuild the nodes on a map and not screw up demos so I assume blockmaps are the same in that respect.

If the same algorithm is used the result will be the same, otherwise not. Currently there are 2 blockmap algorithms in use: MBF's (in PrBoom and Eternity) and ZDoom's. But that doesn't matter. ZDoom doesn't even care about demo compatibility between versions and for the other 2 ports the output will be the same anyway.

I think it's important the wad contains pre-built data just for that reason, even if it does seem like extra bloat.

Guess what? If a port supports omitting the lump and the tools responsible do so as well it won't be included. It really is just bloat.

I suggest after the map format is set to have a very low memory usage, fast compression format- probably basic zip- it should help the filesize considerably, especially on large wads. of course, this is a long way off, so I'm just throwing this out.

Graf Zahl said:

Why not do a text based format? That would make things much easier, eliminate most of the need for version control and also makes many of the awkward multi-format handling unnecessary. And if a database for property was created it could be ensured that if a feature is supported it is done the same way everywhere.

There are so many problems with a text-based format, though. Number one problem is specifying yet another input language, and this time one that people actually agree on. I would personally want it to use a syntax compatible with libConfuse, but that's not really any good for zdoom which uses the Raven SC parser for all its scripts.

Also, going to text-based IO for maps would pretty much eliminate a very important tenet of the project, and that is gaining editor utility support. As is, DOOM Builder would need relatively moderate modifications to handle a third map format. Asking CodeImp to add a new language parser (and keep in mind most of his code is in Visual Basic, too) is probably not going to get very far.

Finally, there's the issue of efficiency. The non-specific fields in my binary specification could easily be mapped to packed structures. Text doesn't map to anything, and you need a dual-loop finite state automaton parser to deal with it.

Graf Zahl said:

Most importantly, remove and mention of 'implementation specific' from the draft. Every known feature that should be supported should have a unique representation in the data. The moment you leave the specific use of flags and other properties to each port's developer we will quickly get the same mess as we have now. This should be avoided in the planning stages already.

The goal of the spec being to get several major ports (or at LEAST zdoom) to agree with the standard required these details to be left out. When specials and flags are specified, it goes a bit beyond a map format and starts to dictate how ports can implement their features. Let's see some of the problems with this, piece by piece.

Graf Zahl said:

- The thing flags word should clearly assign a specific function to each bit. If 32 is not enough extend it to 64 bits. If you want you can reserve some flags for port-specific use.

How is it possible to achieve a common denominator when there's no limit on what ports can add, and no firm overlap for what they already support? Example: 3 bits being dedicated to control the specific class appearance of items in Hexen seems a waste of 3 bits otherwise, and isn't going to help me personally when I add generalized player class support to Eternity. I added translation support to the format in order to hopefully cover this kind of thing. If a map says its preferred translation is Hexen, then you understand its flags field is in Hexen format. It requires more translation/conversion code, but certainly not much more than zdoom is already using.

Graf Zahl said:

- The specials for things and lines should clearly differentiate between Doom-format and Hexen-format. If the records themselves are the same size for both I recommend against using the presence of BEHAVIOR as the deciding factor. And Hexen-type specials should be standardized so that every implementation must adhere to the same name and function for each number. For Doom-format it is too late to do that but for Hexen it is not.

Eternity looks at things quite differently when it comes to support of Hexen-style map specials, and the ones I've already begun to implement are not strictly Hexen/zdoom compatible because I disagree with being bound to limitations of the Hexen map format -- that's mainly the point of this proposal. For an example, take a look at the parameterized linedefs documented in the Eternity editing reference. By using ExtraData, it is possible to use these specials in DOOM -- but it is not convenient, precisely because you need ExtraData.

Eternity also implements a more general trigger system than Hexen. Player/monster/projective activatability statuses are de-wed from the impact/push/walk/etc activatability statuses, and the one-sidedness, repeatability, and other properties are also capable of being specified independently. To require me to switch over to the more limited Hexen idea of PLAYERIMPACT instead PLAYER | IMPACT would be a step in the wrong direction.

Unlike zdoom, I don't even have the luxury of combining normal line types with the Generalized/Parameterized line system, because my old line types must have certain quirks, limitations, and special cases required for compatibility, but that shouldn't be present in the latter system. If I had to combine these systems, I would be facing a rewrite of tens of thousands of lines of code, possible sacrifice of compatibility, and a lot of annoyances.

Using bits of the special to mean certain things is certainly a tempting idea, however it excludes use of a significant range of linedef special numbers and introduces more issues that will agitate cross-port compatibility.

Graf Zahl said:

where 'identifier' is a 4 character token that names the property (and should be registered by the commitee you suggest) to be defined and 'value' is the value being assigned (the format depends on the property.) These are easy to skip in the loading process and avoid having to define extended versions of the various records for each new feature.

I'm a little puzzled by this suggestion because it is extremely close to the mechanism I already suggest for implementation-dependent fields (meaning specifed by one port or another and not an official part of the standard).

Graf Zahl said:

One thing that can be omitted IMO is the BLOCKMAP lump. Even for the largest maps I know the internal blockmap builders need only fractions of a second to build it so there really is no need to waste space for it. AFAIK the only port which doesn't have a BLOCKMAP builder included is Legacy so just for that I see no need to implement it.

Regarding nodes, there's some room to save space. Most of the properties of a seg can be calculated at load time. There is no need to store the angle and the offset. Both values can be quickly retrieved from the linedef they belong to. Reading these from the lump only makes them susceptible to bugs in the node builders. (ZDBSP and WARM definitely have problems with the offset value.) Even the direction is easily calculated by checking the distance to the linedef's vertices so you can cut down a seg's size from 24 bytes to 12 bytes without sacrificing any information (and as we all know, segs can become by far the largest resource in a WAD.)

Again, however, these requirements/changes begin to dictate port features and would require changes to all implementing ports to calculate an appropriate-size blockmap or to calculate missing seg data. While the original reason for adding that seg data (speed) is less of a concern now, it's still convenient to have it ready on map load. My format aimed to require the most significant changes to implementing ports be that of widening their internal data type fields, which seemed a good goal to me, though I may be mistaken.

RjY said:

Yeah I think a text file based format is better than a binary one since even though it is a higher overhead it's easier for humans to debug. Also plenty of tools already exist for text that would require the writing of specific utilities for binary formats - ever wished you could grep or diff or patch wad data? There's plenty of free text parsers around too (has anybody made an xml format for wads? or do you hate xml on principle? :) )

I don't know about missing out the blockmap since, although the port building it itself is feasible, slight differences in algorithms between engines or even between different versions of the same engine could play merry havoc with demo compatibility. I mean you can't rebuild the nodes on a map and not screw up demos so I assume blockmaps are the same in that respect. I think it's important the wad contains pre-built data just for that reason, even if it does seem like extra bloat.

The qualities of text editing that make it good also make it bad -- there are no WYSIWYG editors that can read and write text-format maps, so while it is convenient to be able to modify some things by hand, it is horrible to have to specify EVERYTHING by hand. Consider if you took a Gothic DM map and converted it from binary to text by looking at it in an editor and transcribing every structure in the map. I'll talk to you again in 3 years when you're finished ;)

While XML would be a logical choice for a textual map format, consider that the smallest XML parsers are still larger than the entire source of any DOOM port. It's like swatting a fly with a nuclear bomb.

TinyXML is pretty small. But I really dislike XML, it can be excellent for document formats, and lousy for other stuff. It's horrible to read and edit in a normal text editor.

Two other initial thoughts about the spec (haven't read it closely yet) : REJECT and BLOCKMAP should be optional (allow empty lump). Especially reject, which is usually a big waste of space, and modifying a port to detect an empty lump is trivial. Secondly, for Things there should be a real 'Z' value instead of height-off-floor, to support multiple floors. Maybe have a magic value to mean 'AUTO' (allow easier conversion from DOOM format).

Lastly, I've thought quite a lot in the past about how a new map format could be successfully introduced. It simply won't happen without the editors adding it (CodeImp, Deep). This discussion is futile without their involvement.

I actually prefer the Eternity approach of ExtraData -- keep the standard map format the same and provide a way of adding extra data to it (name/value pairs in a text lump). I don't like Eternity's actual implementation, what it calls MapInfo is different to what zdoom/legacy/hexen call MAPINFO. Creating a good standard way of doing it would be bear more fruit imho.

Ajapted said:
TinyXML is pretty small. But I really dislike XML, it can be excellent for document formats, and lousy for other stuff. It's horrible to read and edit in a normal text editor.

For a map format there is no need for XML's complexity. Simple blocks of data like

{
property1 = value1;
property2 = value2;
}

are enough and don't require a complex parser.

Two other initial thoughts about the spec (haven't read it closely yet) : REJECT and BLOCKMAP should be optional (allow empty lump). Especially reject, which is usually a big waste of space, and modifying a port to detect an empty lump is trivial.

Many ports already support that. But including the option into the spec might serve as motivation for the other ports which still insist on having this data in the WAD to change it.

Secondly, for Things there should be a real 'Z' value instead of height-off-floor, to support multiple floors. Maybe have a magic value to mean 'AUTO' (allow easier conversion from DOOM format).

Sorry, no. The current system with relative values allows changing a sector's height without adjusting any thing in the map. If you have absolute thing heights you'd have to adjust all of them each time the map is changed. There's also no problem with 3D-floors. You just have to use the 3D-floor's offset from the real floor.

Lastly, I've thought quite a lot in the past about how a new map format could be successfully introduced. It simply won't happen without the editors adding it (CodeImp, Deep). This discussion is futile without their involvement.

And the editor developers won't do anything unless something is supported by various source ports. Someone has to start. But frankly, I'm more interested in creating a more flexible format rather than expanding the limits. So far I have seen one single map that became huge enough that the limits became a concern and what's worse, DoomBuilder is already reaching its performance limit with that map.

I actually prefer the Eternity approach of ExtraData -- keep the standard map format the same and provide a way of adding extra data to it (name/value pairs in a text lump).

I can tell you from several discussion about using advanced features in the ZDoom forum that mappers would like to have an easy way to use this stuff directly in an editor rather than having to bother with a script or ExtraData lump or whatever to use them. So if this is the way to go a standard has to be defined that can be implemented by an editor. (Admittedly this is probably easier to achieve than establishing a new map format.)

I don't like Eternity's actual implementation, what it calls MapInfo is different to what zdoom/legacy/hexen call MAPINFO. Creating a good standard way of doing it would be bear more fruit imho.[/B]

I never looked to closely at the code. Are you telling that Eternity uses a lump called MAPINFO that contains completely different data than the ZDoom/Legacy/Hexen format?

Quasar said:

Consider if you took a Gothic DM map and converted it from binary to text by looking at it in an editor and transcribing every structure in the map. I'll talk to you again in 3 years when you're finished ;)

Well, quite :) It'd be quicker to write something to do the conversion automatically, and indeed I think somebody would have to write such a program for this proposal to have any chance of success.

Graf Zahl said:

If you have absolute thing heights you'd have to adjust all of them each time the map is changed.

Aren't the 'z' values in hexen format Things absolute?

I can tell you from several discussion about using advanced features in the ZDoom forum that mappers would like to have an easy way to use this stuff directly in an editor rather than having to bother with a script or ExtraData lump or whatever to use them.

Ideally the editors will support the new system, whatever form it takes. A binary format requires an editor or no one can use it. At least with an Extradata-like system, it can be edited by hand until the editors catch up.

I never looked to closely at the code. Are you telling that Eternity uses a lump called MAPINFO that contains completely different data than the ZDoom/Legacy/Hexen format? [/b]

No, text stored in the map header lump (MAP01 etc) is called MapInfo, and uses a different format from the MAPINFO lump but similar keywords. That's my understanding anyway, Quasar correct me if I'm wrong.

Ajapted said:

Aren't the 'z' values in hexen format Things absolute?

No. They are either relative to the floor or ceiling. For normal things they specify the z-offset from the sector's floor and for things with SPAWNCEILING the negative offset from the sector's ceiling.

Ideally the editors will support the new system, whatever form it takes. A binary format requires an editor or no one can use it. At least with an Extradata-like system, it can be edited by hand until the editors catch up.

True. But to catch up the most important options must be properly defined in the spec. As long as the spec talks about 'implementation dependent' there's no chance of any adoption by editing tools.

Taking a closer look at the spec, I noticed that the extension mechanism is almost exactly the same as the chunk system used in PNG, IFF and other formats.

So I suggest making all the modified lumps use the chunk system! Give each lump a standard magic value (to distinguish from normal Doom), followed directly by any number of PNG-like chunks. Using a single magic value is easier to read/write. My suggestion is "xMAP" followed by 0x891AFE1A followed by 0x00000000.

Each chunk is: 4-byte ID (alphanumeric), 4-byte length, and the data (padded to a 4-byte alignment). The 4-byte ID makes the FMTVERSION fields unnecessary. Instead of extending each record individually, a port that wants to add it's own stuff would use a custom chunk type.

Do you really think that's a good idea? I can tell you right now that this will get no editor support. How about something really simple:

The normal map data remains unaltered and all additional properties are stored in the map header as a text file.

Then you could specify:

Sector 666
{
    color_red=255;
    color_green=128;
    ceilinglightlevel=255;
}

and so on. The big advantage of such an approach would be that an editor doesn't have any need to understand all the properties as long as it gets a list of the supported ones and their types. This kind of data could also be used with the existing map format so that the new one only becomes necessary for large maps.

If we're talking Binary + Extendable, then the PNG-like chunk system is a proven solution.

I do like your simple approach, it's basically ExtraData with the data in the map header. The snag is that Legacy and Eternity (and others?) are already using the map header for stuff. It needs some research to check if the new data can co-exist. Also hexen puts something into the map header (a version string IIRC), do you know if that is needed by the hexen engine?

Edit: forgot to say: we won't know what will get editor support until the editor guys tell us.

Graf Zahl said:

So far I have seen one single map that became huge enough that the limits became a concern and what's worse, DoomBuilder is already reaching its performance limit with that map.

What map is this?

I'll implement it to SLADE, if anything gets decided :P

I'd certainly prefer Graf's text-based format, or anything text based really. As he said, text compresses just as well as (or even better than) binary data. Not to mention file sizes don't quite matter as much now as they used to...

Ajapted said:

I do like your simple approach, it's basically ExtraData with the data in the map header. The snag is that Legacy and Eternity (and others?) are already using the map header for stuff. It needs some research to check if the new data can co-exist. Also hexen puts something into the map header (a version string IIRC), do you know if that is needed by the hexen engine?

Yes, that could be a problem. But it could be implemented the same way so that both data can coexist. The SMMU-style header already knows about sections so that mechanism can be expanded.

+1 for text

ravage said:

What map is this?

KDIZD Map04.

Of course it also depends on the machine you are using. A modern system is probably fast enough for it.

If a text-based format can be supported by editors, I am not opposed, but I think restricting it to the map header could be a mistake as many legacy map utilities (including wintex) do not properly recognize map header lumps with data inside them -- or worse, they strip it out when the wad is resaved. This is why Eternity supports global MapInfo too -- I have all but deprecated map header MapInfo and I heavily discourage its use in the documentation. Just having support for it in 2 or 3 editors wouldn't be enough, since those editors aren't also WAD tools. We need something that WAD tools, even older ones that still have important uses, can deal with easily. For that, I suggest a standard mechanism for pointing an editor to an external lump that can hold the same data. This is how Eternity handles ExtraData (btw, please do not confuse ExtraData & MapInfo, they are two totally different things. ExtraData is more or less what is already being discussed here -- a text script for map format extension).

The ONLY reason I'm not happy with ExtraData is precisely because it's more difficult to edit. You must remember what record numbers you have assigned to what lines, sectors, or things, and then go to the text script and give them the properties they need in an environment that is impoverished as far as DOOM editing support is concerned -- a text editor.

Quasar said:

The ONLY reason I'm not happy with ExtraData is precisely because it's more difficult to edit. You must remember what record numbers you have assigned to what lines, sectors, or things, and then go to the text script and give them the properties they need in an environment that is impoverished as far as DOOM editing support is concerned -- a text editor.

Is there any method to include this data somewhere in the map so that it doesn't screw up older WAD tools? The biggest problem I am seeing is that it gets overlooked accidentally if it isn't placed somewhere in the map data itself. So the header is out and most of the other standard lumps as well. Adding another lump to the map data will also confuse the older tools that don't know about it so we are again left with an unsatisfying situation.

Maybe the simplest method is to somehow put it into the THINGS lump. Sure, that would make it incompatible with editors that don't know about it but so would a completely new map format.
But at least it would allow the map data to be handled as one by lump management tools which IMO is more important than being able to load the map into every editor available.

Having this data completely separated from the map in a different lump would make editor support considerably harder unless all the WAD tools were adjusted.

Graf Zahl said:

Maybe the simplest method is to somehow put it into the THINGS lump.

Or maybe abuse the SIDEDEFS lump. Have dummy sidedefs at the end and use the 8-byte texture names like this:

upper: record ref e.g. "s666" for Sector 666
lower: field ref e.g. "ceillite"
middle: field value e.g. "255"

Put some special values in x/y offsets (like 0x891A,0x2BEF) to distingush them from normal sidedefs. Set sector number to #0 to be safe.

The big downside is that these sidedef lumps will be massive (can't compress them).