Pending UDMF Revisions

[Graf Zahl]

entryway said:

Why octal and hexadecimal are allowed? Is there any sense for map data?

No. But it's just a part of strtol so support for it comes for free normally.

[Quasar]

Skipping over the complete rehash of the argument that took place during my original binary map format suggestion and again during the UDMF definition process...

Here is the regexp for double values in the current UDMF spec:

[+-]?[0-9]+"."[0-9]*([eE][+-]?[0-9]+)?

Whereas these are the regexp's for values accepted by strtod, which corresponds to the syntax supported in virtually all C-syntax languages (C, C++, Java, JavaScript, etc):

[+-]?[0-9]+([Ee][+-]?[0-9]+) -OR- 
[+-]?[0-9]*"."[0-9]+([Ee][+-]?[0-9]+)? -OR-
[+-]?[0-9]+"."[0-9]*([Ee][+-]?[0-9]+)?

(Note that within the languages themselves, an (f|F)? is also allowed at the end, but this is irrelevant to UDMF as it specifies that all floating point literals are at least double precision).
The main differences are summarized as follows:

• A C float need not contain a decimal point, *if* it has an exponent specifier. UDMF floats require a decimal point unconditionally.
  
• A C float need not begin with a digit before the decimal point, so long as it has one or more digits after a decimal point. UDMF floats *must* begin with a digit after any present sign.
  

I guess when you boil it down, there's nothing that needs to necessarily be added to the UDMF spec in this department. The problem is that if strtod or sscanf are used to parse the file, non-compliant values may be accepted, and if certain fprintf specifiers are used to generate the file, non-compliant values could be created inadvertently.

I am aware that CodeImp is currently idle, however it might be possible for boris or someone else to apply changes to the DB2 codebase for a hypothetical UDMF 1.2. I'm not even certain if CodeImp implemented the escape characters that were previously discussed or not, and Eternity is going to *require* those in order to store Aeon inside maps, since the earlier heredoc suggestion was universally rejected.

As far as the other two grammatical problems, those are outright errors/oversights I made while writing the grammar, and should be fixed regardless. The presence of those errors runs counter to the spirit of the specification, which should at least to this extent be considered superior to its accidental letter.

As for the precision problem, the question is one of whether or not strtod and other C/C++ stdio/string/stdlib functions perform over-stepping rounding operations when dealing with double values, and how one determines the precision at which a double value should be printed.

Note that when handed to printf, a default precision is usually assumed if none is specified. This means that simply dumping the value into a file with a specifier like %g is likely to round it to a place such as 6 digits right of the decimal point.

For example the default value for Eternity's s_highgain console variable is nominally 0.8; however, 0.8 is *not* a representable value in IEEE floating point, so you'll find the value in memory is actually something like 0.80000000004. However, passing the variable to printf using a simple %g will result in the value 0.8 being displayed. Depending on the crusty old C library to be internally consistent with regards to rounding various different float values is scary, to say the least. What if vertex values become artifically inconsistent with the BSP? Precision is unnecessarily lost and the mathmatically correct behavior of the BSP could be at risk.

I also find defining UDMF's behavior for floating point in terms of how it converts to fixed point an odd requirement, considering that floating point vertex values were specified in UDMF precisely to circumvent the limitations of fixed-point math, by request of several other people.

Eternity itself has the ability to deal with floating point vertex values already; it simply has no current means to get these from a file - they may only be generated internally through dynaseg translation or rotation. For purposes of the Cardboard renderer, only the floating point vertex coordinates are actually used. The fixed-point versions exist for the benefit of gamecode only.

[Graf Zahl]

Quasar said:

• A C float need not contain a decimal point, *if* it has an exponent specifier. UDMF floats require a decimal point unconditionally.
  
• A C float need not begin with a digit before the decimal point, so long as it has one or more digits after a decimal point. UDMF floats *must* begin with a digit after any present sign.
  

I guess when you boil it down, there's nothing that needs to necessarily be added to the UDMF spec in this department. The problem is that if strtod or sscanf are used to parse the file, non-compliant values may be accepted, and if certain fprintf specifiers are used to generate the file, non-compliant values could be created inadvertently.

Expanding the spec to allow the other forms would be better, I think. There are tools which may inevitably create non spec-compliant values depending on the libraries they use.

I am aware that CodeImp is currently idle, however it might be possible for boris or someone else to apply changes to the DB2 codebase for a hypothetical UDMF 1.2. I'm not even certain if CodeImp implemented the escape characters that were previously discussed or not, and Eternity is going to *require* those in order to store Aeon inside maps, since the earlier heredoc suggestion was universally rejected.

I don't think the escape characters are a problem by themselves. Much more important will be that the editor can handle the input of multi-line text and be able to convert it to something proper for UDMF storage. Heredoc would pose the same problem, btw.

I also find defining UDMF's behavior for floating point in terms of how it converts to fixed point an odd requirement, considering that floating point vertex values were specified in UDMF precisely to circumvent the limitations of fixed-point math, by request of several other people.

It's irrelevant what's the precise spec but to make life easier for tool implementors there should be something that specifies how precision should be checked, regardless whether it's checking the fixed point representation or any other cutoff. Fixed point (meaning that 16 bit fractional precision needs to be preserved) seems natural though. It's not too many fractional digits but also not too few.

[DaniJ]

Why is it so convenient to ignore what people like myself have said about UDMF? Instead of ignoring the issue, surely it would have been better to discuss how to extend the spec so that it can be used. Not very universal is it?

This is the last I'm saying on the matter (hurrah! I hear you cry) because frankly I'm sick of the blinkered mindsets of those involved in defining the spec.

@Graf We just have an awful lot of additional geometric and especially lighting data. To give you an idea, our bias lighting model will compute a lightgrid that encompasses the entire loaded map. This lightgrid is a quadtree of interpolated sector lighting values. Even in small maps like those in the IWADs, just this data structure alone can easily reach 5mb and there is only one of them atm. In larger maps the size of just this one structure can very quickly sky rocket to five times that and takes quite a while to compute. Ideally we need to be able to fracture stuff like this into quickly-uploadable fragments and that means using a similar form in the archived map data.

[esselfortium]

The BIAS lightgrid sounds like a good candidate for just storing in its own format in a separate lump, though, for the same reason that UDMF doesn't try to do text-formatted nodes.

[DaniJ]

That is precisely my point. UDMF has been designed to meet the needs of but two ports; namely Eternity and Zdoom. The format of core data structures merely reinforces that.

UDMF simply is not a universal format. Its basically exactly the same as the original DOOM format with the ability to add a few additional properties to things like SIDEDEFS.

[Gez]

It was designed to easily add additional properties to the standard elements from the user-editable lumps in the original format (THINGS, VERTEXES, SIDEDEFS, LINEDEFS and SECTORS). Anything additional goes to its own lumps -- so for example ACS still goes in BEHAVIOR; for Strife conversation scripts a DIALOGUE lump has been reserved by ZDoom, etc.

A port that does not need to add new properties to standard elements, but instead want to add brand new things, has of course less reason to adopt it. But this doesn't mean they couldn't make use of it anyway. The removal of limits (especially for the Hexen-side of things, with all its single-byte values) would still be useful even without new editing features.

[DaniJ]

Well yes, we could make use of UDMF for adding new properties to the old DOOM map format (yes we do add a lot of new properties to those elements). However we would still have to go through exactly the same conversion process that we already do with DOOM/Hexen format maps to get them into a format suitable for our needs. So, given we already do this and would still have to do it with UDMF - where is the benefit? We already have a textual definition system for this.

[Graf Zahl]

DaniJ said:

That is precisely my point. UDMF has been designed to meet the needs of but two ports; namely Eternity and Zdoom. The format of core data structures merely reinforces that.

UDMF simply is not a universal format. Its basically exactly the same as the original DOOM format with the ability to add a few additional properties to things like SIDEDEFS.

No, you are missing the point completely. Gez already stated the basics but the intention of the UDMF TEXTMAP lump was never to handle elements that don't fall into the common categories. And even if it was possible it wouldn't be a good idea because editing tool support hinges on it so this part of a map needs to be defined in a way that makes writing an editor for it possible. In fact this was the most crucial aspect of all. Without it the format would have been a dead end right from the start.
But since obviously that's not the whole story, UDMF was designed in a way that makes adding new lumps to a map straightforward, easy and robust because the extent of a map is determined by the ENDMAP lump. So all the stuff you are talking about logically goes into other lumps that are to be inserted between TEXTMAP and ENDMAP, just like prebuilt nodes and scripts. You can put there whatever you want in whatever format you want.
Your BIAS lightgrid would be a perfect example of something that would be inserted into a map this way.

So in fact the format is as 'universal' as theoretically possible. Even if you add all your stuff in the proper way you still have a valid UDMF map whose core can be read by any other port as well.

[DaniJ]

I don't think I'm missing the point at all. You aren't understanding what I'm saying. We already have systems in place for extending the map format with new properties in a human-editable definition system. We can already put those in a ZIP alongside the original DOOM/Hexen format map data. So the prospect of adding UDMF support doesn't gain us anything. It merely makes map loading slower because we now have to parse a text file for the core data structures instead.

[Gez]

DaniJ said:

I don't think I'm missing the point at all. You aren't understanding what I'm saying. We already have systems in place for extending the map format with new properties in a human-editable definition system. We can already put those in a ZIP alongside the original DOOM/Hexen format map data. So the prospect of adding UDMF support doesn't gain us anything. It merely makes map loading slower because we now have to parse a text file for the core data structures instead.

Eternity has something like that, too, it's called ExtraData, and it was not enough to stop Quasar from wanting something more practical. ZDoom has something roughly like that as well, through map translators, but nobody is using it for ZDoom modding, just to make compatibility patch (like the Caverns of Darkness one).

And a big issue is editor support. Having additional properties be available just through a tab in DB2 is handy and makes the mappers happy. Having to note numbers and go edit a text lump in a text or lump editor at the same time you're editing the map in a map editor is cumbersome. Just as cumbersome as having to make a script for it.


Storing brand-new data in the map data, even if it would make sense in a way, has two crucial problems:
1. The map editors would not support them without being updated.
2. Custom editors just for this data would have to support map data (and not screw it up!).

With UDMF, these problems are not present. ZDoom or Eternity could add a thousand new thing properties overnight, and DB2 wouldn't need to be updated to support them. And programs such as ACC or USDC do not need to mess with the TEXTMAP data.

[DaniJ]

I understand what you and Graf are saying but being able to use DoomBuilder2 to change the color of sector lighting (for example) does not sound like much benefit to me, considering you can do the same in Doomsday itself with WYSIWYG style feedback, and have it combine them into the map for you.

UDMF literally gains us nothing.

[Graf Zahl]

Gez said:

And a big issue is editor support. Having additional properties be available just through a tab in DB2 is handy and makes the mappers happy. Having to note numbers and go edit a text lump in a text or lump editor at the same time you're editing the map in a map editor is cumbersome. Just as cumbersome as having to make a script for it.

In fact that was the main issue. You can cook up whatever you want but if it means you have to take out a text editor and write control data referencing the core map by hand you have already lost. With UDMF it can all be done directly in the editor, even for fields the editor knows nothing about and the strong association between entities and their properties ensures that nothing gets messed up.

If you write a tool to generate this data putting it into an extraneous control lump is just another cause for errors because the lump is not packaged with the map and can get lost. Being able to insert such a control lump into the map in a well defined way is certainly more robust.

Last but not least, what's the problem with parsing the TEXTMAP lump? It takes more time? Great! Does it really matter if the user has to wait 3.2 instead of 3.0 seconds? I don't think so. If the TEXTMAP parsing would introduce a significant overhead things would be different. But the time spent for that is so minimal that it's hardly an issue. Even in ZDoom the texture precaching alone takes longer.

[DaniJ]

UDMF is great for Zdoom and Eternity because it means they can share maps in a format each can extend in the same way, promoting future feature parity. I do get it.

I really am going to walk away from this now because we aren't getting anywhere, as it is very clear that this is the only usage scenario being considered.

[Graf Zahl]

If we had done what you apparently had in mind the format would be dead due to lack of editor support.

As I said before the core TEXTMAP lump was never about adding new entities, just opening the existing ones up to future expansions.

It was also not about sharing maps. The goal was to create something that

• allows to assign arbitrary new properties to the existing entities in a well defined fashion
  
• allows adding arbitrary new data that's not defined in the core spec to a map without breaking things
  
• has support by a commonly used map editor.
  

So I'm sorry but I really can't see why it's not 'for you'. The fact that you can cleanly package your new data into the map as opposed to alongside the map should be worth something, shouldn't it?

[DaniJ]

I thought I'd already said that Doomsday will combine the new data generated by it and/or users into the map itself. So the sidecar-file benefit is a non-issue.

The core data structures in a UDMF format map are exactly the same as the original DOOM format. This is useless to us. We would have to go through the same process we already do. So no benefit there either.

Users would have to constantly jump back and forward between DB2 and Doomsday because the editor can't visualize what we want to add to the format without extensive changes (to both format and editor). DB2 is Windows only so thats half our users who can't benefit from this. Not to mention we'd have to go through the whole conversion process every single time.

Most likely we will implement UDMF importing in Doomsday just so we can load maps in that format but other than that its no use to us.

[Graf Zahl]

DaniJ said:

The core data structures in a UDMF format map are exactly the same as the original DOOM format.

Of course they are. With the significant difference that they can be expanded. They were never supposed to be more because everything beyond that would be port specific and therefore subject to port specific specs.

And whether this is 'of no use', as you say is debatable. For many of your visual stuff probably not.

However, keep in mind that UDMF's base spec has floating point precision vertices, extended line trigger semantics, 32 bit args for Hexen line specials and things, separate spawn flags for all 5 skill levels and ZDoom even adds more such things. And please don't tell me that such features are 'of no use'. If you purely concern yourself with advanced visuals you may be correct but that's hardly the full story behind a new map format.

BTW, since you continue to post here I can only assume that it somehow bothers you that you find UDMF of little use to you. So where was your feedback when it was discussed? Back then when you had the chance to contribute, the best we got was 'Doomsday is currently not interested'. So this left Quasar, CodeImp and me to discuss and define UDMF. Any surprise that other ports' needs didn't get the attention you would have liked? I still think that it's good and open enough to be expanded accordingly though.

[printz]

Graf Zahl said:

As I said before the core TEXTMAP lump was never about adding new entities, just opening the existing ones up to future expansions.

That would stick Doom in the 2.5D world with its five components. Ports intending to expand the engine as they see fit would have little use for UDMF. I guess this is DaniJ's problem.

[Graf Zahl]

printz said:

That would stick Doom in the 2.5D world with its five components.

How is a base spec supposed to go beyond that? It's one of the fundamental properties of Doom and changing this would make all existing editing tools obsolete.

[DavidPH]

printz said:

That would stick Doom in the 2.5D world with its five components.

Except that it doesn't. There is no reason you couldn't use the 5 object types to create true 3D architecture. In some ways you do, 3dmidtex for instance. 3D floors to a slightly lesser degree (it uses a control sector so it's sort of in between).

[DaniJ]

The reason I keep posting here is because you keep inviting a response from me by trying to show how UDMF will do what we want.

UDMF is of interest to me because of the fact its a new format that we'll have to support sooner or later. What you decide to do with the spec itself really doesn't bother me as what ever it ends up being I'll have to write an importer anyway.

I tried offering feedback on the spec at the time but every time I offered input on how it would affect its suitability for our use, I was ignored. Presumably because it was taking it away from what Zdoom and Eternity needed.

Graf Zahl said:

However, keep in mind that UDMF's base spec has floating point precision vertices, extended line trigger semantics, 32 bit args for Hexen line specials and things, separate spawn flags for all 5 skill levels and ZDoom even adds more such things.

How is any of that useful to me? Vertices do not need floating point precision as all the existing editors do not make use of it. Line trigger semantics, skill levels, flags etc, etc are already expanded in our DOOM/Hexen format importer.

As printz surmised; the existing DOOM-derived tool chain is already obsolete for what we want to do with map authoring.

[Gez]

Discussion kinda went on a loop I feel. Both sides of the argument restating their position and neither seeming to actually understand why the other actually has a different view point.

[DaniJ]

Agreed. Thats why I keep trying to exit the conversation.

[Graf Zahl]

DaniJ said:

I tried offering feedback on the spec at the time but every time I offered input on how it would affect its suitability for our use, I was ignored. Presumably because it was taking it away from what Zdoom and Eternity needed.

I can't remember you giving any concrete info about your needs. At least the two main threads that started UDMF don't contain anything regarding the format itself, except for one suggestion making the format more complex than necessary (I'm talking about creating sub-blocks for line special parameters.)

[DaniJ]

If the UDMF specification had been expanded in such a way it would have allowed us to use it similarly to DED - thus allowing users to use DB2 to set the same properties they already can via DED. In such a scenario we wouldn't actually use UDMF for the map data. Instead it would be used entirely for setting the "extra" properties. As it stands UDMF offers less functionality than we already have.

I used the line trigger example as it was something I thought the others in the discussion could relate to.

However this is all academic now.

[printz]

DavidPH said:

Except that it doesn't. There is no reason you couldn't use the 5 object types to create true 3D architecture. In some ways you do, 3dmidtex for instance. 3D floors to a slightly lesser degree (it uses a control sector so it's sort of in between).

The more complicated they become, the more maintenance sectors/linedefs/things they require, instead of being defined from more direct means. But I wasn't really talking about those -- they were already designed to fit in the 2.5D world. I was talking more about the case when the port is more ambitious on revolutioning the world, adding new rendering methods where the default five wouldn't suffice (a brushed or meshed world combined with the traditional one could be an example).

Graf Zahl said:

and changing this would make all existing editing tools obsolete.

I think compatibility may be solved by adding the new types in the menu under a "custom structures" group (alongside things, linedefs and so on), and when activated the user would be able to access each object from a table listbox. Some of the fields could be user-defined to mean crossreferences or coordinates, in order to give some visual aid in the editor.

If custom types aren't prototyped at the start of TEXTMAP, then Doom Builder 2 would detect them and save them in an accompanying data file to the WAD. If they are, then all the better, DB2 would know what new structures to load.

Printz out.

[Gez]

printz said:

The more complicated they become, the more maintenance sectors/linedefs/things they require, instead of being defined from more direct means. But I wasn't really talking about those -- they were already designed to fit in the 2.5D world. I was talking more about the case when the port is more ambitious on revolutioning the world, adding new rendering methods where the default five wouldn't suffice (a brushed or meshed world combined with the traditional one could be an example).

But then you're better off using a brand-new map format. Which you can base on anything you fancy, be it a collection of binary entries like Doom, a text file, or chunky data à la PNG. I could understand, for example, implementing the Quake map format in such a port. Then you'd already have GTK Radiant or Quark or whatever other Quake-engine tools exist to make maps for your new format.

But it would be hard to convince other ports to follow you here. With UDMF's design, you may not gain anything by supporting it, but it's relatively easy. The hassle of having a parser (all ports already have parsers anyway, even if it's only for configuration files or DeHackEd patches) is insignificant compared to the hassle of implementing scene graph- or brush-based logic in the physics and rendering engines... All Doom ports have support for the 2.5D map structure, otherwise they wouldn't be Doom ports. UDMF is therefore a non-intrusive feature: maybe it will not bring anything new to the port, but at least it will not force anything new either.

Something with the intention of being an universal compromise cannot afford to be "ambitious on revolutioning the world" unless it wants to only be universal in the rejection it inspires.

[RestlessRodent]

UDMF needs port specific areas and ALL editors supporting must not drop things it does not understand, only leave them alone or warn about them. I'm not sure of this but I believe CodeImp said that if there's anything Doom Builder 2 finds it does not know is that it won't save them.

On light of this, UDMF shouldn't use integer values for whatever, but perhaps hashes for sectors things and such. It can be human readable but it can also be human understandable, using text instead of binary is pretty useless if no human will ever modify it directly. On large UDMF maps, nobody is going to remember every id of every single sector or thing if they are tweaking maps (or making them) outside of an editor. Instead of a telport line being tagged to "84" it can be instead tagged to "ZombieTeleportTrapRoom" and the corresponding sector will have that tag.

entryway said:

There is no difference between
if (*(int*)buf == seg_length) seg.length = GetIntValue(buf+4);

Ouch, that code is bad and you shouldn't be doing that!

Why? Because:
1. If buf is not aligned correctly on an architecture which requires alignment, you just SIGBUSed your program (ARM, PPC, and Sparc).
2. May violate strict aliasing screwing up the optimizer due to differences in size.
3. buf may have garbage at the end causing values to be changed incorrectly when cast to int.
4. buf might not be in native byte order.

[kb1]

This discussion is silly (in reference to above discussion about binary formats, and embracing UDMF vs. something custom)
...UDMF *is* a universal format, not just a ZDoom or Eternity format. The fact that I can open a file, and immediately parse and interpret it into at least a partially working map, is unbelievably convenient.

Even if a port or editor completely doesn't understand certain contained properties, those properties will persist, and be carried along throughout the lifetime of an object. A save in an editor will preserve the values, even values it completely doesn't understand, and there's just no other reasonable way to accomplish that.

And, yes, it could have been in a binary format, and still been extensible. Quite a bit harder to verify, but, sure it's possible. In fact, someone could, down the road, create a spec for a 'compiled UDMF' format. But that's really more of an afterthought, and not really part of this discussion.

The whole point of the format is to allow arbitrary port-specific and yet-unknown properties to be added to normal Doom objects, in a manner that ensures that editors can carry them along with all of the basic map data, even if the editor doesn't understand them, and, to allow ports to open and load the maps, choosing which custom features to support, and which to ignore.

In fact, a port can *know* that it's providing less than 100% support for the map in question, and alert the player.

Perhaps there's some justification to the idea of adding a 'blob' datatype that could store arbitrary binary data, which would, because UDMF is a text format, have to be written as hex. This 'blob' type could hold the additional geometry, lighting, whatever.

But, I distinctly remember Quasar inviting anyone and everyone to review and suggest modifications to the pending spec...

Besides, there's nothing stopping anyone from using a UDMF field to point to an arbitrary lump within the wad, which could be loaded in whole or in part, on demand, etc. That 'pointer' would be carried alongaide all the normal map data too. Very similar to textures/flats.

All in all, UDMF has a few slight shortcomings, but, out of the box, it adds a new level of flexibility that had to be 'hacked in' before. And, it's currently the best thing going. It's in everyone's interest to support it, and, if need be, collaborate to design needed revisions that differ as little as possible from the original spec.

I spent the first five years of my "doom career" changing the map, texture, image, wad formats, etc. to support all the new ideas I wanted to add to Doom. To the point that it would no longer load standard wads. But, that was ok, because I also built map, texture, image, and wad converters. Yay. I had it all working rather well. And it had some really cool features that were, as of then, in no other port. And, it was fun to play.

But, there were problems:
1. I could convert a standard map to my format, but it couldn't be converted back *because my format was custom!!*.
2. If anyone wanted to play my maps, they had to use my port, *because my format was custom!!*.
3. Custom converters and editors were required, *because my format was custom!!*.

So, I've spent a lot of time reverting back all of those changes, because, clearly, that was the wrong approach to take. Compatibility is EVERYTHING. It's taken me so long to get to this point that I still don't have a demonstrable port (but, I'm getting there).

The lesson I learned here is that changing something 'just because you can' is not necessarily the best approach. Embracing a format thats very philosophy is to attempt to be as compatible, flexible, and easy to create as possible, even if it's not a 100% perfect fit, makes a lot of sense.

Make it fit, whatever that takes, because everything else is bullshit, in some way or another.

[Graf Zahl]

DaniJ said:

I used the line trigger example as it was something I thought the others in the discussion could relate to.

Well, all I can say then is that it was a bad example because it only demonstrated adding more complexity for nothing.

GhostlyDeath said:

I'm not sure of this but I believe CodeImp said that if there's anything Doom Builder 2 finds it does not know is that it won't save them.

If that's the case DB2's implementation is not compliant. Any tool that can write out maps again *MUST* preserve unknown fields or it'd have to be considered destructive. But I don't think DB2 is doing that. Unknown fields will be added to the UDMF property list, I think, where they can be edited in raw form.

On light of this, UDMF shouldn't use integer values for whatever, but perhaps hashes for sectors things and such. It can be human readable but it can also be human understandable, using text instead of binary is pretty useless if no human will ever modify it directly. On large UDMF maps, nobody is going to remember every id of every single sector or thing if they are tweaking maps (or making them) outside of an editor. Instead of a telport line being tagged to "84" it can be instead tagged to "ZombieTeleportTrapRoom" and the corresponding sector will have that tag.

Technically you are correct but this falls into the category of 'intrusive' feature. It needs support in all ports supporting UDMF (in case of ZDoom it'd be a trivial addition) but worse, it'd need support in editors, too, and the main concern when defining the format was to ensure we get editor support. Yes, some compromises had to be made, that don't make much sense if you wanted to define a 'perfect' format, but it was the only way to get it off the ground.

@printz (about breaking the 2.5D barrier):

To be clear, that was well beyond the scope of UDMF. Such a map format does not make sense as a 'universal Doom map format' because it'd be incompatible with all Doom ports. If you want true 3D you need a true 3D engine which Doom is not. This alone renders the whole idea pointless.

kb1 said:

The whole point of the format is to allow arbitrary port-specific and yet-unknown properties to be added to normal Doom objects, in a manner that ensures that editors can carry them along with all of the basic map data, even if the editor doesn't understand them, and, to allow ports to open and load the maps, choosing which custom features to support, and which to ignore.

Precisely. And this alone means that the suggestion Entryway made above about using integer tags for the fields is pointless. Why?
Imagine an editor made for ZDoom UDMF spec 1.1 opening a map made for spec 1.10. Now what happens? Right: It encounters some unknown fields and has to abort because it a) does not know what this field means and b) doesn't even know what type that field is! It just cannot parse beyond it. Result: The format is no longer universal! And even if the type issue could be resolved, it still wouldn't know what the field means and present it to the user in a way that won't be of any use. Not what we want.

Perhaps there's some justification to the idea of adding a 'blob' datatype that could store arbitrary binary data, which would, because UDMF is a text format, have to be written as hex. This 'blob' type could hold the additional geometry, lighting, whatever.

Well you gave the answer yourself already:

Besides, there's nothing stopping anyone from using a UDMF field to point to an arbitrary lump within the wad, which could be loaded in whole or in part, on demand, etc. That 'pointer' would be carried alongaide all the normal map data too. Very similar to textures/flats.

;)
Better, you can let it point to an arbitrary lump within the Map itself! and prevent naming conflicts from ever happening.

The lesson I learned here is that changing something 'just because you can' is not necessarily the best approach. Embracing a format thats very philosophy is to attempt to be as compatible, flexible, and easy to create as possible, even if it's not a 100% perfect fit, makes a lot of sense.

Make it fit, whatever that takes, because everything else is bullshit, in some way or another.

Well said. I had an engine like that, too, several years ago, but got tired of it because nothing ran on it without being modified. That's not a solution.