Intellectual debts

[Ralphis]

VGA said:

That was a common map standard. It is not the same as engine features.

Support for UDMF is most definitely an engine feature.

[VGA]

When someone proposes a commmon format, he wants everyone to adopt it. When someone is working on an engine feature, he wants to be the first to implement it :D

[Ralphis]

The point that I was making was that if the individual who carried the torch on the format hasn't even implemented it into his own engine after 8 years, how long should other engines have to wait to implement something like portals that also are not 100% finished? Eternity has a lot of really awesome features and it also has a number of half-finished ones as well.

If the ZDoom project followed the suggestion that they should wait so that EE doesn't get the wind taken out of its sails, how long should that wait be? How long have linked portals been incomplete in EE?

Doom is open source and, as long as everything is being properly done within the licenses, it's fair game.

[Gez]

kb1 said:

In the case of ACS, that was adopted after it was quite mature, and let's be honest - it wasn't exactly a novel feature that made ZDoom stand out.

You sure about that? ZDoom adopting the Hexen map format -- and with it ACS -- was the one thing that set it apart from all other Doom ports, and if you look at early ZDoom wads -- the stuff by Kurt Kessler, Cyb, Tommie Quick, Bryant and Kevin Robinson, etc. -- you'll find ACS prominently.

[Graf Zahl]

VGA said:

When someone proposes a commmon format, he wants everyone to adopt it. When someone is working on an engine feature, he wants to be the first to implement it :D

Actually, when someone proposes a common format I'd assume some personal interest to get it off the ground - not let the competing engine run away with it - which is precisely what happened here.

[Quasar]

Graf Zahl said:

Actually, when someone proposes a common format I'd assume some personal interest to get it off the ground - not let the competing engine run away with it - which is precisely what happened here.

Listen to my interview and find out why it took so long to get UDMF into EE. It was not my top priority and it had a million and one prerequisites to clear. Those were all done as of the last official release.

[Coraline]

UDMF has also been a point of interest for me, I just haven't had the time since we've been updating other stuff in 3DGE. Perhaps one day I'll get around to it!

I totally think sharing code is a win-win. I'm glad to see this issue resolved, Eternity and GZDoom are both great ports!

[printz]

I wish UDMF wasn't so ridiculously verbose, and that it was closer to DECORATE. Just the "<block_name> { <key> <value> <key> <value> }" kind of syntax. No wasteful visual noise caused by =, ", ; and such.

[Graf Zahl]

What's the problem with that 'visual noise'? The main reason it was done like this was because the format is supposed to be robust with well defined delimiters that in some cases allows resynchronization of a parser and continue after an error (i.e. if parsing a keyword or value goes wrong, skip over to the next '=' or ';' and resume.)
And parsing strings without quotation marks is simply not doable. How else will you be able to find spaces etc. in there without breaking the parser. Even DECORATE requires this.

Or is it some problems with libConfuse that make this hard to process?

[andrewj]

printz said:

I wish UDMF wasn't so ridiculously verbose, and that it was closer to DECORATE. Just the "<block_name> { <key> <value> <key> <value> }" kind of syntax. No wasteful visual noise caused by =, ", ; and such.

Huh wtf? It is a format that is primary designed to be read and written by software, not by humans, so a few extra characters does not matter much.

Be thankful that it is not based on XML!

[Graf Zahl]

To further explain why this is not noise but absolutely necessary for a robust format, look no further than SNDINFO what can happen if you leave out that 'visual noise':

SNDINFO looks like this:


logicalsoundname reallumpname
logicalsoundname reallumpname
logicalsoundname reallumpname
logicalsoundname reallumpname
logicalsoundname reallumpname

And now imagine what happens if some stray line contains

logicalsoundname reallumpname 1

The parser has no chance to catch this problem and not report an error and from the 1 onwards will parse the following data incorrectly until some special command comes along, leading either to an error message in the wrong place (which most likely is totally unclear) or result in undefined behavior with strange error messages later.

And that's what'd happen here, too if there were no delimiters (since UDMF parsers are REQUIRED to ignore unknown keys!)

[printz]

Meh, I guess you're right. I didn't use libConfuse in my Eternity UDMF branch, I just wrote my own interpretation, which I also had to rewrite in the process to be fast enough. My complaint is based on the fact that I need to read and pass through "useless" tokens while looking for the key and value of a block, but it doesn't matter that much I guess.

I'd rather have a free and good UDMF library, than expect every other Doom programmer to pass through this "ordeal", especially when it has some edges to take care of:
- it must be fast. If you use intuitive hash tables, it takes too much to load a crazy level. My current Eternity implementation uses simple C structs for the intermediary UDMF blocks (NOT the in-game linedefs and so on) whose data is stored using key string->offsetof() maps.
- some fields are required. If they're missing of the invalid type, the map setup should stop.
- some fields are position sensitive ("namespace" must be the first), while others are freely roaming (all the block fields). Blocks are definitely positional with others having the same name.

You suggested ZDBSP as a source. Is it generic enough? I'm tempted to start a generic TEXTMAP reading library, with emphasis on fast and extensible.

andrewj said:

Be thankful that it is not based on XML!

There are XML parsers out there, though I guess if I want cross-platform free ones, there are fewer. I guess I should be thankful in case none is fast enough to load huge maps!

[Graf Zahl]

printz said:

- it must be fast. If you use intuitive hash tables, it takes too much to load a crazy level. My current Eternity implementation uses simple C structs for the intermediary UDMF blocks (NOT the in-game linedefs and so on) whose data is stored using key string->offsetof() maps.

That's something I do not really understand why it could be a problem. In ZDoom I use the global name table for finding the keywords and turning them into indices, then using those for one large switch/case block and even the largest stuff around requires less than half a second to parse the entire level - and that time includes all internal allocations and reallocations - there was never any conscious effort to optimize. Of course, a level that needs half a second to parse will require several seconds afterward to load all its resources, so that half second is more or less irrelevant. The code is pretty much in the same state as my very first draft. So what made it slow on your side?

BTW, looking at your code, I didn't see any mechanism to reject Eternity-namespace keys in the base namespaces. They should not be allowed to work there.

[printz]

Initially I used Eternity's MetaTable objects to store all blocks. After storing everything in a big EHashTable of Collections of MetaTables, I started loading the level itself. Each world item would read a MetaTable by name-addressing each key, checking for the existence and checking its runtime class... Maybe I could have cached the key hash names or not, but in any case a converted BTSX E1M1 would load MUCH slower than its vanilla format counterpart.

Do you create the world items as soon as you read the blocks? I.e. as soon as a linedef{} block ends, a new game linedef is completed?

Graf Zahl said:

BTW, looking at your code, I didn't see any mechanism to reject Eternity-namespace keys in the base namespaces. They should not be allowed to work there.

All known fields in all compatible specifications are loaded as-is into the intermediary block structs. It's when the setup creates the game world items (e.g. in E_LoadUDMFLineDefs) that wrong namespace fields are ignored.

I also (I'm telling you because you commented a few months ago) reduced the influence of TEXTMAP to a simple map format whose simple existence won't affect gameplay. Basically the "namespace" value won't be used anywhere past the setup stage (instead I'll use whatever values are setup by the current IWAD and game-mode-info). And "id" and "arg0" will be mapped precisely to their line_t equivalents, which means that Boom line specials will still use "id" for their target sectors. This is at Quasar's request and I agree it makes it easier for me.

[Graf Zahl]

printz said:

Do you create the world items as soon as you read the blocks? I.e. as soon as a linedef{} block ends, a new game linedef is completed?

Yes and no, actually. The Things block gets parsed into an intermediate structure which later gets passed to P_SpawnMapThing.
All the other data is created in its final format directly but since the final data structures need the size they get placed into a dynamic array and after everything is done, the final structures get allocated and everything transferred with a memcpy.

printz said:

I also (I'm telling you because you commented a few months ago) reduced the influence of TEXTMAP to a simple map format whose simple existence won't affect gameplay.

Normally this is the same in ZDoom, but we had to make one exception because for historic reasons one particularly badly motivated 'feature' needs to be the default - that'd be what the engine calls 'lax monster activation'. It's something that predates my involvement in ZDoom (I would never have let this allowed to get into production), but essentially it means that monsters can activate some linedefs that are flagged 'player can activate' only. With a new format we had the chance to set a sane default. Of course this can be changed through MAPINFO. (I sincerely hope that you do not try to copy that crap for Hexen format maps when the time comes... ;))

printz said:

Basically the "namespace" value won't be used anywhere past the setup stage (instead I'll use whatever values are setup by the current IWAD and game-mode-info).

Yes, same with ZDoom. The namespace is only used for deciding which fields are valid and whether some values have to be restricted (e.g. restricting line specials for "Doom" namespace to what MBF has and null-ing everything else.)

printz said:

And "id" and "arg0" will be mapped precisely to their line_t equivalents, which means that Boom line specials will still use "id" for their target sectors. This is at Quasar's request and I agree it makes it easier for me.

I'm afraid I do not understand what you mean here. There never was an issue with sectors, only with specials affecting other lines. So what precisely so you plan?

[printz]

P_SetupLevel has some objects which depend on others. Which means that there has to be a "deswizzling" of pointers (convert indices to pointers) after all map items are loaded.

Graf Zahl said:

I'm afraid I do not understand what you mean here. There never was an issue with sectors, only with specials affecting other lines. So what precisely so you plan?

Fine. The lines teleporting to other lines, or making other lines transparent, as inherited from Boom, will still just use "id". If other advanced line types are created in Eternity, they'll probably be parameterized and stop using "id" for everything, both source and target.

[Graf Zahl]

printz said:

P_SetupLevel has some objects which depend on others. Which means that there has to be a "deswizzling" of pointers (convert indices to pointers) after all map items are loaded.

Obviously during parsing, there's no way to store pointers, so they get stored as indices which later get resolved. For sidedefs, they also need decompressing to work as intended in ZDoom. But aside from such things, the intermediate data is stored in the same data structures as the final version (i.e. line_t, side_t, sector_t.)

printz said:

Fine. The lines teleporting to other lines, or making other lines transparent, as inherited from Boom, will still just use "id". If other advanced line types are created in Eternity, they'll probably be parameterized and stop using "id" for everything, both source and target.

Ok, but keep in mind that although this is ok for Doom format maps, it is a violation of the UDMF spec where it is made very clear how these are supposed to work. To quote:

Unlike traditional Doom maps UDMF makes a clear distinction between a line's
ID and the parameter which identifies the object the line's special is
supposed to affect.

The id will be used to identify this line, and arg0 will be used to identify
the line or sector that this line is to affect or otherwise reference, i.e.
it is effectively a parameter to the line's special.

Boom used the linedef's tag for both due to lack of other options so in
order to ensure compatibility any map converter converting maps for the
Doom/Heretic/Strife namespaces must store the linedef's tag field as both
the id and as arg0 in each line. The default value of the id field under
these namespaces is defined as 0 rather than -1.

I fought hard to get this part in such an explicit and verbose form approved by Quasar (in fact it was more than half of the entire discussion to get sorted out!) so I's very disappointing to see this being ignored for some questionable convenience issue after all. If you are concerned about demo compatibility issues or similar things, you should at least create a duplicate that actually does what the spec says.

Seriously, what's the problem with it? There are no maps depending on it remaining unchanged, and the actual change is rather trivial (i.e. change the FindLineFromTag function to use the real argument to the special and make sure it gets copied for Doom format maps.

You have to be aware that, should you really implement it like this, the map will not be compatible with other engines that implement UDMF with spec conformance!

[Quasar]

Graf Zahl said:

Seriously, what's the problem with it? There are no maps depending on it remaining unchanged, and the actual change is rather trivial (i.e. change the FindLineFromTag function to use the real argument to the special and make sure it gets copied for Doom format maps.

You have to be aware that, should you really implement it like this, the map will not be compatible with other engines that implement UDMF with spec conformance!

id and arg0 being different from each other in a Doom, Heretic, or Strife UDMF namespace map is illegal. Those maps do not have separate ids for lines therefore a compliant map from those namespaces should have id == arg0 for all lines. So I fail to see what difference it makes.

I regret if there was ever a misinterpretation that the UDMF standard was adding features to the vanilla namespaces - that was the opposite of its intent.

[Graf Zahl]

Quasar said:

id and arg0 being different from each other in a Doom, Heretic, or Strife UDMF namespace map is illegal. Those maps do not have separate ids for lines therefore a compliant map from those namespaces should have id == arg0 for all lines. So I fail to see what difference it makes.

No, that's absolutely not correct.

We had this discussion 8 years ago and finally we agreed on the opposite because it's a relatively minor change. The spec says nothing about both having to be equal. I quoted what we agreed upon verbatim in my last post, and the wording is quite explicit (which I made sure of back then

Just read the old thread:

https://www.doomworld.com/vb/source-ports/43207-udmf-v1-0-rfc/


Well, I guess I'll have to check myself how much of a problem this would actually constitute for Eternity and maybe do a pull request if I manage to do something about it. To be honest, I think it can't be that difficult.

EDIT: After checking the entire codebase I don't even think there's any real work needed at all.
To solve this problem all that needs to be done is:
- initialize args[0] with tag for Doom format maps.
- change the loop which initializes translucency in P_LoadLinedefs2 to use arg[0] instead of tag.
- Change P_FindLineFromLineTag to use line->args[0] instead of line->tag as the value to look for.
- Define some clean semantics how Extradata initializes these variables. But considering that the only released map with Extradata doesn't depend in any way on this that shouldn't be a roadblock. My suggestion would be: ED::tag initially goes to both line_t::tag and line_t::args[0]; if ED::id is present it overrides line_t::tag, and if ED::args are specified they override line_t::args[0].

Unless I grossly missed something that should make the entire thing compliant to the spec as written.

[Quasar]

Easy or not I still don't understand how it makes a difference. If all compliant translators must place tag in both id and arg0, how can a compliant map end up with id != arg0 in the standard namespaces?

Only if somebody hand-edits one to do so. Why would they ever do that? Is this an actual feature that ZDoom is allowing to be exploited in compatibility namespaces? If so, why?

See, unlike ZDoom, we don't convert everything internally into a Hexen format. DOOM specials must still behave like DOOM specials. And that means when a line's tag is cleared, *the line's tag must be cleared* - anything else is a gigantic threat to demo and map compatibility. I cannot just rewire parts of EE to use a different field as tag for DOOM maps w/o carefully considering the possible impacts.

This has absolutely no bearing on specials that would be used in EE's "Eternity" UDMF namespace, since they can be written to work properly using the action bindings system from the get-go, and utilize parameterized specials natively.

[Graf Zahl]

Quasar said:

Easy or not I still don't understand how it makes a difference. If all compliant translators must place tag in both id and arg0, how can a compliant map end up with id != arg0 in the standard namespaces?

What makes you think that such maps can only be made by 'compliant translators'?

What about something somebody built in an editor, using different values for both fields? The spec's wording not only allows that but also very clearly states how it should behave.

This

linedef // 30
{
v1 = 43;
v2 = 41;
sidefront = 51;
special = 272;
arg1 = 2;
id = 5;
}

linedef // 31
{
v1 = 53;
v2 = 51;
sidefront = 41;
special = 272;
arg1 = 1;
id = 5;
}

linedef // 32
{
v1 = 54;
v2 = 55;
sidefront = 42;
special = 218;
arg1 = 5;
}

would be a perfectly valid setup for defining two sky transfers and then using a single scroller to affect both lines through the separate ID - all perfectly within what the spec describes.

[RestlessRodent]

I have not read the code (despite Eternity being license compatible with ReMooD, nice that you are on the GPLv3 by the way), but to implement linked portals I would do something similar to Boom's teleport line and do some BSP rendering trickery to look into that region. Then after that I would create a limited set of how close one area is with relation to a portal (so monsters and bots choose that route instead). In fact, since I am doing amajor rewrite into Java I can actually more easily code in the assumption that there will be linked portals.

UDMF

I have a plan to implement UDMF when I work out the map formats. However a problem for me is that there is going to be a gian hole in the set of compatible and available features if wesley has no interest in adding UDMF support.

Personally I would have preferred UDMF have a kind of modulated set of features (feature flags) that one could enable and use in many source ports. Say a map uses 3D Floors, you could just have an enabler which says that they are available. However with UDMF you can right now only choose to not use them at all, or use a specific port (which another port which can use 3D Floors might not understand at all). I have been thinking about a "FeatureBits" namespace of sorts which can derive its options from a global namespace group. Then if a map uses a given feature and it is not available, then you warn about it or just not play the map.

[Graf Zahl]

You are possibly correct.

I think a system, where a base namespace is specified, followed by

'implements = "featuregroup"'

would be a good idea to modularize the spec somewhat.
I'll have to check ZSBSP at least to see if it can handle unknown keys in the global section - that implementation is somewhat old, and may not have the capability yet.


Alternatively do it like:

namespace = "Doom; implements 3DFloors, Portals, Slopes";

or whatever syntax is acceptable.
But since this is a new namespace, you'd be free to define the spec as you like, especially the supported feature groups. This could be done without revising the base spec.

[RestlessRodent]

Graf Zahl said:

namespace = "Doom; implements 3DFloors, Portals, Slopes";

Having it in the namespace name might get rather ugly especially when there is the potential of having a large number of "features". Right now at the start there would probably be a handful, but say 4 years down the line users might want to use 40+. What I am thinking of is something such as this:

featurebits // Whatever special naming scheme is required
{
    common_3dfloors = true;
    common_portals = true;
    remood_vmcode = true;
    gzdoom_stereoscopiclensflares = true;
    // Anything else defaults to false
}

If I am reading the spec right it says "Compliant parsers will ignore all unknown keywords in global assignments, block-level assignments, and block headers."

Then (hopefully) define or come to agreements what the actual stuff means and then put it in source port documentation. Would suppose for an example with 3D floors it would follow the legacy line numberings (I am assuming ZDoom/GZDoom use the same ones for compatibility) for Doom while going for the ZDoom/GZDoom ones when using Hexen. The common stuff would just be sane values, while port specific stuff could get crazy.

An alternative from line IDs (in the event of line number collisions) would be a string based "feature_special" (i.e. "gr teleport" or similar with a not horrible syntax) with a unique special of say 32767/65535/..., but that could be something else. Then the ID based specials could sort of be broken away from, I am going to do that in ReMooD anyway (specials get translated to an internal representation) as I have always hated that `if (0 != (special & floor_raise))` with the power of lambda.

But yes, I would be very much for modularization because as it stands right now maps are just in one bucket instead of many. UDMF gets defeated in some areas when the only common ground is the Doom/Heretic/Hexen namespaces. As authors of ports we can agree within a modular namespace instead of having some common strict subset of features in a fixed namespace that would require ratification everytime something wants to be added (i.e. "CommonGround20160221", "CommonGround20160227", etc.).

Since no actual "ReMooD" namespace exists yet, I would rather just have a single modular namespace. Otherwise UDMF is just a new level format that has a common format but is actually very specific to ports.

Mind you that implementing support for this will be far easier where I am going (in ReMooD) compared to where I was. In the end, it will be Doom Legacy only in spirit.

[Graf Zahl]

Such a change would most likely require bumping the version number, including all current implementations. And wide consensus among implementing ports.
Not that I am against such a change. It's probably best to think about it NOW, where other ports start to implement UDMF as well.

In hindsight I have to agree that the entire namespace thing is doomed to fail as it stands, if more ports get support for it and each one defines its own, incompatible namespace. That scenario will inevitably result in something highly incompatible where we do not have a 'universal' map format anymore. And the ports cannot even pick the required features out of what is present. Not a good solution.

I'd like to see something like OpenGL's extension mechanism:
The base namespace defines a certain behavior, with all non-compliant features DISABLED - even if said feature doesn't require any specific UDMF keys.

Like, for example, if you got the Doom namespace, Eternity's portals are not enabled, they only become active after adding an 'eternity_portals' feature flag. And that's not just mean the UDMF keys but also the respective line specials.

Of course if we go that route we also need to be aware of the pitfalls.
In GZDoom I fully implemented Eternity's flat and horizon portals because all they needed was a simple function that used existing code as workers. In ZDoom this stuff is not present and seeing how things currently develop, probably never will be available. So if such a feature that Eternity and GZDoom share, but ZDoom does not, lumping it together under one feature bit with other features which all 3 ports share (e.g. the Eternity skybox definition method) may quickly result in situations where one port can no longer safely determine if the map may work or not.

So if we are not careful, we might end up with something that's ultimately way too hard to maintain by the mappers because there's far too many feature bits.

[printz]

After some discussion on IRC, I realized that Quasar's UDMF goals may be more complex than simply porting the Doom format + ExtraData to a unified TEXTMAP lump. I assumed I would be able to finish the work without him if I went the easy route, since everything is already out there (most of ExtraData maps nicely to whatever ZDoom already has). But he wants to reduce some of the complexity that has gathered from the previous inherited ports, so I think it's up to him to think up the namespace. He may delegate boring (but clear enough) tasks to me.

[Graf Zahl]

Hm...

I still think defining some namespace that exposes all the common features between Eternity and ZDoom might be a good idea. Of course with the current mishmash of Doom-format and Hexen-format features in Extradata that can become quite a problem (well, maybe not, I already expanded ZDoom's XLAT to remap Hexen format specials from Extradata, expanding that to UDMF wouldn't cost me much effort.)

So if this means wider support of Hexen format features it'd ultimately help everybody (and probably render the Doom-format namespaces completely unused... ;))

I'd be interested to know where this 'reducing complexity' would lead.

[Gez]

Tangentially relevant, but I think it was a mistake not to do for thing, line, and sector types what had been done to flags and give them a name.

Compare

special = portal_horizon;

with

special = 1234;

Maybe there could be something like this:

extensions
{
    portal_horizon = 1234;
}

Then this would tell the engine both that there's an extension it needs to handle, and it'd prevent ambiguity and conflicts (e.g. line type 281 is 3DMidTex_MoveWithFloor in Eternity, but Solid 3D Floor with Shadow in Doom Legacy). Engines that implement an extended namespace (the extensions block would be illegal in basic Doom, Heretic, etc. namespaces) would need to have a way to remap extended line/thing/sector types as requested by the extensions block.

It wouldn't be too hard for the map editor to write the extension block based on the configuration being used and the line/sector/thing types actually used in the map; basically what's needed implementation wise for editors is flagging types if types are standard or extended and the internal name to use for extended types. (Or just the presence of an extended name other than an empty string could be enough to flag a type as extended.)

[Graf Zahl]

Gez said:

Tangentially relevant, but I think it was a mistake not to do for thing, line, and sector types what had been done to flags and give them a name.

Compare

special = portal_horizon;

with

special = 1234;

Absolutely correct. But then we wouldn't have UDMF now.

Of course with the line specials we'd be faced with the major problem that we have two mutually exclusive set of specials that could muck up the entire thing. Naming all those Doom-type specials would add a lot of cruft. But going Hexen-format only would leave some ports like RemooD behind.
Another issue is actor names. ZDoom and Eternity use entirely different names so some translation table from a supposed common standard will be needed.
And yet another minefield is the sector types where we sadly missed the chance to take the Boom bitfields out of the equation right away.

So, if you ask me, we should do UDMF 2.0 and actually FIX all those mistakes, instead of doctoring around on the symptoms. I think the format is sufficiently established by now that the supporting editors will go along.
Of course that still doesn't solve the problem of feature groups where two ports only commonly implement some parts but divert in other parts of the same feature group

[MaxED]

Graf Zahl said:

UDMF 2.0

Can it also include array definition syntax, so user arrays can be also defined in a map editor?..