UDMF v0.99 Specification Draft (aka TEXTMAP)

[Graf Zahl]

Quasar said:

Without defining these standard target ports, this is exactly the sort of situation we'll face. Since each port has a different idea of how to unify namespaces, unless you also support translating the original special sets, then it won't be possible to, for example, load UAC_DEAD.WAD, save it in UDMF format, and have the resulting file be just as universally usable as the original was. If the only available target ports were ZDoom or Eternity, it would by necessity contain different special numbers :) If it is "Doom" on the other hand, then we may both use our already-existing code for "Doom" gamemode maps to load it.

If we can define the standard namespaces "Doom", "Heretic", "Hexen" and "Strife" I'd consider the problem solved.

BTW, can we agree that the standard for Doom should include BOOM specials, or do we have to make BOOM a separate target? I can't see any advantage to that, as the two are strictly non-conflicting, and the only difference is that BOOM defines values that are undefined under Doom. If a supporting port didn't support BOOM, it would just do with such specials the same thing Doom always did. Ignore them or throw an error (ignoring being preferable obviously).

I have never seen a non-Boom compatible map that conflicted with Boom due to some garbage in some fields. Each map made for just limit removing ports that used non-Doom specials did so by using Boom types accidentally and became essentially a Boom compatible map as a result - so I see absolutely no need to define a separate namespace. The same would apply to Heretic where the current Boom special support in ZDoom does not interfere with any map known to me. I would have to define stripped translators that omit all non-Boom linedef and sector types but that'd be 5 minutes of work so no big deal.

[Quasar]

Right. I'd prefer that they be non-strict. That is, if the standard definitions do not specify something is used (such as a line type like 263 just to name a random one), then the meaning of it is implementation dependent. This will present the fewest problems.

[Graf Zahl]

The reason I was thinking of limiting it is that the Boom linetypes are quasi-standard and no port redefines them (with the sole exception of MBF's 272 sky transfer colliding with Legacy's/SMMU's Fragglescript type. I'm using some crude hacks in GZDoom to resolve that one, btw.)

However, everything beyond that is different between ports so if the map says 'Boom' I'd rather NULL any unknown special than assuming that it might work.

It'd also serve as a motivation for other port authors to define proper namespaces rather than being lazy and stuff all their new extensions in the base thus rendering the maps useless to other ports.

[Graf Zahl]

Just a small thing I just noticed:

Thing coordinates are integers. It may not be useful for normal things to make them floats but some of ZDoom's added stuff certainly could use the added precision, particularly the slope creating things. Wouldn't it make sense to change the specs there? Also, where's the Z-coordinate? There's only X and Y.

[CodeImp]

Definitely floats for thing coordinates!

[Julian Now]

Quasar said:

The special number problem is related in part to using numbers. Since it has been decided that getting every port to agree on a standardized set of names for specials, types, and flags is impossible, it is necessary to instead come to a compromise, in which we have standard spaces, and each port gets its own private playground as well. This keeps the format universal, but not necessarily the interpretation of the data held in it.

You basically end up with a shell and nothing really standardized for the content. That's too bad.

Beside, you hinder interesting possibilities for source ports.

Take the example of an "ice" special. Does it matter what "mode" the port is in ? If a sourceport can handle icy sectors, then it should be able to do so in any game mode. So you could have icy sectors in Doom, Heretic, Strife, whatever. A sourceport that cannot do icy sectors would simply ignore the name (graceful degradation anyone ?).

Of course, it gets really tricky with numbers.

If I understand your namespace system properly, it's basically a source port ID, isn't it ? Maybe with version added ? If you go with numbers, where will they be specified for each namespace ? (I'm thinking about the headache for editors here)

Quasar said:

BEX Flags syntax was rejected as part of the earlier specification process. It was decided that parsing it was too complicated for some implementations to handle,

Well, you can go for what is basically a list of additive flag names separated by spaces. Is it really that complicated to "parse" ?

Quasar said:

and that separating the flags into separate fields is better, even if it results in a larger file size, which the spec considers a non-issue due to the availability of large hard drives, file compression, and broadband internet connections.

Well, of course, size does not matter. (sorry girls ;))

Thanks for the answers, now onto different questions (for my own personal curiosity):
1) Why use the order of lines, sectors, etc to number them and no "id" parameter (size does not matter, does it ?) ? I suppose load-time memory allocation of arrays ?
2) Why use a proprietary text format and now XML (I do a lot of this nowadays and though it is outrageously verbose, it is also very comfortable once you're used to it) ?

Good luck again guys and thanks to all of you for working on something like this :)

[Graf Zahl]

Julian Now said:

You basically end up with a shell and nothing really standardized for the content. That's too bad.

Beside, you hinder interesting possibilities for source ports.

Take the example of an "ice" special. Does it matter what "mode" the port is in ? If a sourceport can handle icy sectors, then it should be able to do so in any game mode. So you could have icy sectors in Doom, Heretic, Strife, whatever. A sourceport that cannot do icy sectors would simply ignore the name (graceful degradation anyone ?).

Of course, it gets really tricky with numbers.

Some compromises have to be made and if this completely deviates from the existing system it would be DOA. I don't have a problem with numbers even though it means that there's 2 completely different sets of specials (Doom and Hexen mode.) Granted, for Eternity it will become a bit more involved but since Quasar is ok with the system as defined I'll let him sort it out himself. ;)

If I understand your namespace system properly, it's basically a source port ID, isn't it ? Maybe with version added ? If you go with numbers, where will they be specified for each namespace ? (I'm thinking about the headache for editors here)

Same as now: Different configs for different ports.

Well, you can go for what is basically a list of additive flag names separated by spaces. Is it really that complicated to "parse" ?

Compared to the current system, yes. The current system simply allows to run a very simple and very generic parser that doesn't understand much beyond 'key = value;' Any syntactic variation will make matters more involved.

1) Why use the order of lines, sectors, etc to number them and no "id" parameter (size does not matter, does it ?) ? I suppose load-time memory allocation of arrays ?

Simple matter: To keep things simple. Internally they are an array so let's just store them in order. Keeps the parser as simple as possible and avoids needless complications in data maintenance. It doesn't matter anyway because the files are not meant to be hand written. The only reason it's text is that it is easier to extend. Once you define a binary structure, adding more fields will become pure chaos.

2) Why use a proprietary text format and now XML (I do a lot of this nowadays and though it is outrageously verbose, it is also very comfortable once you're used to it) ?

For the simple reason that this would mean a dead end. Again simplicity is the key and anything that involves real work will only mean less acceptance. Writing a parser for the current format can be done in a few minutes. For XML you either need a suitable library (which you need to understand first) or if you don't want that you have to bother with the format's verbosity yourself.

[Quasar]

Graf Zahl said:

Just a small thing I just noticed:

Thing coordinates are integers. It may not be useful for normal things to make them floats but some of ZDoom's added stuff certainly could use the added precision, particularly the slope creating things. Wouldn't it make sense to change the specs there? Also, where's the Z-coordinate? There's only X and Y.

Z would be an extension since none of the original map formats contain it. However if we have early agreement that it should be there, I can add it to the 1.0 RFC.

Float coordinates for things do make sense since we have agreed on float vertex coordinates. I did not want to define any coordinates as floating point until I was certain that Doom Builder would be able to handle them. (But even if Doom Builder does not support floating point thing coordinates, it could always just convert the integer coordinates to/from float on map save/load - all integers with 56 or fewer significant bits can be represented perfectly in IEEE double precision floating point).

As for the BOOM thing, you do have a good point. It would probably be a good idea to go up to MBF and then say that's where it cuts off. Anything else should be handled under the port-specific configuration. For example a Doom-format Eternity map with portal line specials such as 281 should be converted to UDMF with Eternity namespace, not UDMF with Doom namespace. I intend to create a reference command-line map converter tool which will try to take account of such things.

[Graf Zahl]

Quasar said:

Z would be an extension since none of the original map formats contain it. However if we have early agreement that it should be there, I can add it to the 1.0 RFC.

Hexen has it so it must be part of the base spec.

[andrewj]

Graf Zahl said:

Hexen has it so it must be part of the base spec.

Hexen doesn't have a true Z coordinate, only a height above the ground, which depends on what sector the thing is in. For 2.5D engines this makes more sense than a true Z coordinate, because the default of zero is going to be used for most things in the map, but in a few situations it isn't ideal, like when placing things on 3D floors.

[Graf Zahl]

It would have to be the same anyway for compatibility.

[andrewj]

It is surely possible to have a true Z coordinate, as the mobj_t struct (or whatever C++ class it evolved to) in the source has a z coordinate, so the engine could just load the value directly. But yeah the point is moot as the pain would outweigh the slight benefit.

[Quasar]

"height" is already in the v0.99 spec. It is not called z precisely because it is not a z coordinate. If you want a true z coordinate in addition to the Hexen height field, you need to specify now before the standard is finalized.

[Graf Zahl]

Ok, I must have overlooked because I was looking for z only. No, we don't need both.

There's one other thing that needs to be addressed though:

      notsingle = <string>; // true = not in SP mode. Default = false.
      notdm     = <string>; // true = not in DM mode. Default = false.
      notcoop   = <string>; // true = not in Coop. Default = false.

Didn't we come to the conclusion that these flags are better positive instead of negative?

[Quasar]

The draft has not been updated.

v1.0 RFC will be out shortly ;) If no comments are generated, it will automatically become the final spec for v1.0.

I am currently adding stuff for Strife support.

[Csonicgo]

for nostalgia's sake we should do the famous "first pwad" map in UDMF if that's possible as an accomplishment.. map.



you know, the one done by hand? heheh.

[Creaphis]

Well, if it's not possible, that wouldn't be saying much for this wad format.

[RjY]

Csonicgo said:

for nostalgia's sake we should do the famous "first pwad" map in UDMF if that's possible as an accomplishment.. map.

you know, the one done by hand? heheh.

For nostalgia's sake I will do this by hand in a text editor Actually bugger that it's too much like hard work

So, first attempt at a conversion of origwad to textmap format. I haven't done much checking. There might be bugs in the conversion, I might have misunderstood the spec, blah blah blah. Let's just shake the tree and see what falls out :)

linedef { v1 = 0; v2 = 1; id = 0; frontside = 0; blocking = true; }
linedef { v1 = 1; v2 = 2; special = 1; id = 0; frontside = 1; backside = 2; twosided = true; }
linedef { v1 = 2; v2 = 3; id = 0; frontside = 0; blocking = true; }
linedef { v1 = 3; v2 = 4; id = 0; frontside = 0; blocking = true; }
linedef { v1 = 4; v2 = 5; id = 0; frontside = 0; blocking = true; }
linedef { v1 = 5; v2 = 0; id = 0; frontside = 0; blocking = true; }
linedef { v1 = 1; v2 = 7; id = 0; frontside = 3; blocking = true; dontpegbottom = true; }
linedef { v1 = 8; v2 = 7; special = 1; id = 0; frontside = 4; backside = 5; twosided = true; }
linedef { v1 = 8; v2 = 2; id = 0; frontside = 3; blocking = true; dontpegbottom = true; }
linedef { v1 = 7; v2 = 6; id = 0; frontside = 6; blocking = true; }
linedef { v1 = 6; v2 = 11; id = 0; frontside = 6; blocking = true; }
linedef { v1 = 11; v2 = 12; id = 0; frontside = 6; blocking = true; }
linedef { v1 = 12; v2 = 13; special = 11; id = 0; frontside = 7; blocking = true; dontpegbottom = true; }
linedef { v1 = 13; v2 = 10; id = 0; frontside = 6; blocking = true; }
linedef { v1 = 10; v2 = 9; id = 0; frontside = 6; blocking = true; }
linedef { v1 = 9; v2 = 8; id = 0; frontside = 6; blocking = true; }

sidedef { sector = 0; texturemiddle = "BROWN1"; }
sidedef { sector = 0; texturetop = "BIGDOOR2"; }
sidedef { sector = 1; }
sidedef { sector = 1; texturemiddle = "DOORTRAK"; }
sidedef { sector = 2; texturetop = "BIGDOOR2"; }
sidedef { sector = 1; }
sidedef { sector = 2; texturemiddle = "STARTAN1"; }
sidedef { sector = 2; texturemiddle = "SW1STRTN"; }

vertex { x = 0; y = 0; }
vertex { x = 0; y = -128; }
vertex { x = 0; y = -256; }
vertex { x = 0; y = -384; }
vertex { x = -384; y = -384; }
vertex { x = -384; y = 0; }
vertex { x = 18; y = 0; }
vertex { x = 18; y = -128; }
vertex { x = 18; y = -256; }
vertex { x = 18; y = -384; }
vertex { x = 402; y = -384; }
vertex { x = 402; y = 0; }
vertex { x = 402; y = -160; }
vertex { x = 402; y = -224; }

sector { floorpic = "FLOOR4_8"; ceilingpic = "CEIL3_5"; lightlevel = 255; }
sector { floorpic = "FLOOR4_8"; ceilingpic = "FLAT20"; lightlevel = 255; }
sector { floorpic = "FLOOR4_8"; ceilingpic = "CEIL3_5"; lightlevel = 255; }

thing { x = -360; y = -192; angle = 0; type = 1; easy = true; normal = true; hard = true; }
thing { x = -32; y = -32; angle = 0; type = 2001; easy = true; normal = true; hard = true; }
thing { x = -32; y = -352; angle = 135; type = 9; easy = true; normal = true; hard = true; }
thing { x = 338; y = -64; angle = 225; type = 3001; easy = true; normal = true; hard = true; }
thing { x = 338; y = -320; angle = 135; type = 3001; easy = true; normal = true; hard = true; }
thing { x = 50; y = -320; angle = 45; type = 3003; easy = true; normal = true; hard = true; ambush = true; }
thing { x = 50; y = -64; angle = 335; type = 3003; easy = true; normal = true; hard = true; ambush = true; }
thing { x = 50; y = -192; angle = 180; type = 3001; easy = true; normal = true; hard = true; ambush = true; }

Note to future readers: this is both buggy - those sectors should have ceiling heights - and obsolete, as udmf1.0 changed a lot of the variable names. Maybe I'll update when the spec is finalised.

[EarthQuake]

Wow, so we're looking at an extremely huge filesize increase because the files are stored in text and not binary. What about the occasional doom wad where a map might be several megabytes. How much data are we talking about in this format? 10 MB? More?

[Quasar]

By far the largest lumps in a DOOM wad are the SEGS and other BSP resources, and those have been intentionally *not* defined as a part of UDMF. Traditional node builders aren't going to understand UDMF maps anyway so it is just as well.

The average length of the thing entries in that example is around 100 bytes. A Hexen mapthing was 20 bytes. So we're looking at something around a 5 times size increase by this rough estimate. So if a map file is around 200 KB *before* BSP, reject, blockmap, and any script lumps it might have, then it would be 1 MB after conversion. But a 200 KB map before any of those data structures is added sounds pretty damn large to me.

[CODOR]

EarthQuake said:
Wow, so we're looking at an extremely huge filesize increase because the files are stored in text and not binary.

On the other hand, there's a lot of redundancy there so they ought to compress really well.

A size comparison (in bytes) of RjY's origwad (origwad.txt) with the original (origwad.pwd, and its original zipfile) using a few common compression formats:

1392 origwad.pwd
 686 origwad.pwd.bz2
 640 origwad.pwd.gz
 800 origwad.pwd.lzo
3008 origwad.txt
 610 origwad.txt.bz2
 558 origwad.txt.gz
 730 origwad.txt.lzo
1202 origwad.zip

Not really a fair comparison since as Quasar mentioned the generated lumps are missing from the text representation. Plus a sample size of one isn't very useful, but in this particular case it shows that the compressed file sizes aren't too different...

[Graf Zahl]

I did some test with node-stripped maps and overall the compressed size of the new format is ca. 120 - 130% of the binary format.

As for nodes, let's see what the node builders will implement.

Right now I only intend to support the compressed GL nodes format in ZDoom to keep things simple. GL nodes only so that the annoying habit of some mappers to provide useless standard nodes for large GZDoom maps will be stopped. ZDoom can use these as well so for large ZDoom maps it's clearly the best option. If any other port wants to implement support for that format I'll be happy to help with the implementation.