Universal Doom Texture Format

Hi, happy holidays!

Firstly, I could not find related threads in a search; sorry if this has been brought up before. Also, sorry if this is an inappropriate forum for this discussion; I found a few threads regarding the UDMF ratification here and I assumed it would be OK. Feel free to move or delete this thread if necessary.

I've recently been implementing the UDMF v1.0 for a tool I'm writing, and it occurred to me that the same kind of abstraction might be useful for texture-related lumps (specifically, TEXTURE1/TEXTURE2/PNAMES).

Here's what I think something like that might look like:

namespace = "Doom";

patch { id = 109; name = "WALL57_2"; }
patch { id = 110; name = "WALL57_3"; }
patch { id = 111; name = "WALL57_4"; }

texture
{
    id = 61; name = "SLADRIP1"; width = 64; height = 128;
    patch { id = 109; xoffset = 0; yoffset = 0; }
}
texture
{
    id = 62; name = "SLADRIP2"; width = 64; height = 128;
    patch { id = 110; xoffset = 0; yoffset = 0; }
}
texture
{
    id = 63; name = "SLADRIP3"; width = 64; height = 128;
    patch { id = 111; xoffset = 0; yoffset = 0; }
}

animation {
    frame { texture-id = 61; /* delay = .300; transition = "instant"; */ }
    frame { texture-id = 62; /* delay = .300; transition = "instant"; */ }
    frame { texture-id = 63; /* delay = .300; transition = "instant"; */ }
}

The most important thing to note is the nested "blocks". I figured this would be more elegant than suffixing an additional id to a keyword (frame00/frame02/etc.) or prefixing keywords to avoid clashing (patch00-id/patch00-xoffset/etc.) -- which would make the code much more verbose and possibly introduce limits on the number of patches/frames (depending on the format of the names). This introduces both incompatibilities and added complexities to existing UDMF v1.0 parsers. Not having to deal with a recursion stack (no nested blocks) in UDMF v1.0 is nice, so I think this is a critical issue.

The idea grew from working with DOOM's animated textures, and wanting a way to remove the restrictions on first/last texture names and such. So, as you can see the animations are texture-agnostic and of course the syntax allows for other properties of the animation to be adjusted while being, as far as I can see, vanilla-compatible.

Another important question is relevance. You'll have to forgive my ignorance, as I'm not very familiar with either Heretic/Strife/Hexen, nor the latest source ports/game mods. So, a unified texture format may be unnecessary, if not impossible. Replies from more knowledgeable people are absolutely appreciated.

Thanks for your time.

I doubt that this will catch on. Unlike previous 'universal' formats text based implementations for texture definitions already exist so supporting this would mean double maintenance for the ports already implementing an alternative format.

Of course, if there's a general interest I wouldn't stand in the way but seeing how UDMF went with only ZDoom implementing it (and Vavoom copying ZDoom's implementation) I think it's obvious where this is heading. So sorry, unless some other port author shows some genuine interest (and by that I mean, starting an actual implementation) this will be DOA.

Also, what's that redundant ID for? I don't see any use for it except complicating matters.

Graf Zahl said:

Of course, if there's a general interest I wouldn't stand in the way but seeing how UDMF went with only ZDoom implementing it (and Vavoom copying ZDoom's implementation) I think it's obvious where this is heading.

That's kind of disingenuous to say considering you already know that UDMF is on the short-term TODO list for Eternity. I am knee-deep in a C++ conversion effort at the moment, which is only one thing on the list of things that should be done before I try to implement UDMF (inventory and weapons are the other two, because they interact with LOCKDEFS, which interact with keys, which need to be redefined in terms of player inventory - and also I am sick and tired of not being able to play Heretic in EE, and these are the only two things stopping it).

So that's the battle plan, and now you know, and so you should not be going around bemoaning the non-implementation of UDMF. As for this suggestion I think it's too early to say whether we would be interested or not, as we don't even support the ZDoom format yet - there is nothing technical in the way of this, I just have not had time to get to it since SoM finished the new texture system 6_9

kickassand said:

Hi, happy holidays!

Firstly, I could not find related threads in a search; sorry if this has been brought up before. Also, sorry if this is an inappropriate forum for this discussion; I found a few threads regarding the UDMF ratification here and I assumed it would be OK. Feel free to move or delete this thread if necessary.

I've recently been implementing the UDMF v1.0 for a tool I'm writing, and it occurred to me that the same kind of abstraction might be useful for texture-related lumps (specifically, TEXTURE1/TEXTURE2/PNAMES).

Here's what I think something like that might look like:

Why couldn't it be instead like:

namespace = "Doom";

texture "SLADRIP1", width = 64, height = 128;
{
    patch "WALL57_2", xoffset = 0, yoffset = 0;
}
texture "SLADRIP2", width = 64, height = 128;
{
    patch "WALL57_3", xoffset = 0, yoffset = 0;
}
texture "SLADRIP3", width = 64, height = 128;
{
    patch "WALL57_4", xoffset = 0, yoffset = 0;
}

animation {
    frame "SLADRIP1", delay = .300, transition = "instant";
    frame "SLADRIP2", delay = .300, transition = "instant";
    frame "SLADRIP3", delay = .300, transition = "instant";
}

A lot simpler.

What is even the point of messing with ID numbers in a text format? You can zap them out entirely.

What is the point of pre-declaring the patches first? Even the old DEUTEX texture format did not do that. It is entirely unneeded: that's a recreation of the PNAMES lookup table whose purpose is to convert ints into strings; and here we have a text format. Use strings instead!

Also, I'm not sure namespacing is needed here.

If you want to work on a possible universal standard, you should read up on this earlier proposal and the ZDoom TEXTURES format. Since you want it to handle animations as well, you should also read up on ANIMDEFS.

Graf Zahl, Gez, thanks for the replies.

Regarding existing text-based formats, I haven't yet done enough research to know if a good/practical way of bringing those together is possible. (This post is followed by some more in-depth repository browsing.)

When looking into UDMF v1.0 (as briefly as I've done) I was mildly surprised not to find a reference implementation somewhere, or some official form of testing software. It appears more tools have implemented it than source ports. Parsers can be tedious, and even the most naive, near bolt-on implementation can help things to get started -- as perhaps Vavoom's reuse of ZDoom's code shows? -- especially when it's verifiably guaranteed to work. I would lean heavily towards developing such a test suite for a texture definition spec, particularly given its likely complexity. (No doubt this is something UDMF would benefit from as well; not to say I haven't seen very nice existing implementations.)

Regarding the "redundant ID", I believe Gez's posted syntax above illustrates your point nicely. Much simpler to work with; feels a little more like XML with texture and patch info as attributes. There is some inconsistency with the names -- like there's an implicit `name=' there or somesuch -- but looks very good. BTW, thank you Gez for the links, bookmarked immediately.

The "implicit name=" is because they just wouldn't work if this was omitted. You could have default values for the offsets, for example, so that if it's 0, 0 you could just omit them entirely. But you can't afford to omit the names.

Quasar said:

That's kind of disingenuous to say considering you already know that UDMF is on the short-term TODO list for Eternity.

I wasn't implying anything. However, you can't deny that 2.5 years after the final spec ZDoom (and derivatives of course) and Vavoom are the only ports that have implemented UDMF.

So with regards to UDTF (or whatever this could be named) things don't look good. The interest is low and as you yourself stated, you are knee-deep in other things and have a long list to work through so where would this fit in?

ZDoom and Eternity would be the only ports who might have use for this and ZDoom already implements an alternative so that leaves us with no port that might be interested.

Quasar said:

I am knee-deep in a C++ conversion effort at the moment, which is only one thing on the list of things that should be done before I try to implement UDMF

Good to hear.

Quasar said:

I think it's too early to say whether we would be interested or not, as we don't even support the ZDoom format yet - there is nothing technical in the way of this, I just have not had time to get to it since SoM finished the new texture system 6_9

Ok, but still, what sense does it make to define yet another texture definition format, especially if it can't do anything the ZDoom format could not support, too? So for me it'd be an exercise in redundancy and it'd make more sense to expand ZDoom's texture format and make that the standard.


Also, one other issue I have with this suggestion here that it's far more verbose so harder to write.

@Gez: What you propose is almost what ZDoom does, just with more unneeded syntax in between.

Taking this

texture "SLADRIP1", width = 64, height = 128;
{
    patch "WALL57_2", xoffset = 0, yoffset = 0;
}
texture "SLADRIP2", width = 64, height = 128;
{
    patch "WALL57_3", xoffset = 0, yoffset = 0;
}
texture "SLADRIP3", width = 64, height = 128;
{
    patch "WALL57_4", xoffset = 0, yoffset = 0;
}

and removing the verbosity we end up with:

texture "SLADRIP1", 64, 128
{
    patch "WALL57_2", 0, 0
}
texture "SLADRIP2", 64, 128
{
    patch "WALL57_3", 0, 0
}
texture "SLADRIP3", 64, 128
{
    patch "WALL57_4", 0, 0
}

And you know what? We just got a subset of ZDoom's TEXTURES format!

I agree that a cross port standard for defining TEXTURES (in a textual format) would be useful and although I dislike the brevity of the syntax used; it is quite sufficient.

It is my intention to support the TEXTURES method of texture definition in Doomsday. However we presently have a long list of higher-priority work and lets face it, there isn't really a huge immediate benefit in adding support for this.

Graf Zahl, I've been reading ZDoom's TEXTURES lump format and there are some really nice engine features there (I need to get ZDoom on my speedier machine). Regarding the syntax, it seems very similar to Gez's version as you pointed out.

I do have an issue with non-named arguments/attributes/options though. I can appreciate being terse with things that are mandatory and immediately obvious (names). In this case, however, the implied order of the width and height integers, and the offset integers -- while perhaps semi-intuitive -- is something I'd like to avoid; partly because when humans are editing/maintaining the text, it's prone to error, and partly because inconsistency with regards to parsing the text creates more complex code: mandatory arguments/attributes/options are still just arguments/attributes/options, and are in general easier parsed like any other (IMHO. I've written a few BASIC parsers, and the inconsistency there creates some rather stiff code -- to say the least!).

DaniJ, thanks for your reply. I am learning that something like this is a low priority across the board. If there is any demand at all, it has to start somewhere, though. I will look into the Doomsday engine ASAP.

Quasar said:

Graf Zahl said:
[...] but seeing how UDMF went with only ZDoom implementing it (and Vavoom copying ZDoom's implementation) I think it's obvious where this is heading.

That's kind of disingenuous to say considering you already know that UDMF is on the short-term TODO list for Eternity

Yes, and also, UDMF is far more than what is needed for engines that don't extend Doom's gameplay much beyond what Boom does already, and thus don't need an unlimited number of new fields (just 32-bit sidedef, vertex references, etc.)

kickassand said:

I do have an issue with non-named arguments/attributes/options though. I can appreciate being terse with things that are mandatory and immediately obvious (names). In this case, however, the implied order of the width and height integers, and the offset integers -- while perhaps semi-intuitive -- is something I'd like to avoid; partly because when humans are editing/maintaining the text, it's prone to error, and partly because inconsistency with regards to parsing the text creates more complex code: mandatory arguments/attributes/options are still just arguments/attributes/options, and are in general easier parsed like any other (IMHO. I've written a few BASIC parsers, and the inconsistency there creates some rather stiff code -- to say the least!).

This essentially boils down to 'I don't like the format so let's do another one with the same feature set and a different syntax'. I'm sorry but I consider that a waste of time.

RjY said:

Yes, and also, UDMF is far more than what is needed for engines that don't extend Doom's gameplay much beyond what Boom does already, and thus don't need an unlimited number of new fields (just 32-bit sidedef, vertex references, etc.)

Even then, considering that PrBoom+ projects like Sunder or Vela Pax are flirting with the binary map limits on the number of sidedefs, lindedefs, vertices, etc.; these Boom engines would benefit from UDMF even if only for its limit-removing aspect.

They'd still have to make 2 significant changes for UDMF:

- allow different spawn filters for all skill levels
- separate the tag parameter from the line ID

But these really are the only requirements for a working UDMF implementation aside from the actual parser.

Gez said:

Even then, considering that PrBoom+ projects like Sunder or Vela Pax are flirting with the binary map limits on the number of sidedefs, lindedefs, vertices, etc.; these Boom engines would benefit from UDMF even if only for its limit-removing aspect.

Yes but I am thoroughly unmotivated to write patches for support of even basic Doom-namespace UDMF just for limit removal in PrBoom, when I know all that it really needs is for a handful of the fields in the map structs to go from 16 to 32 bits. And to shove a magic number on the start of the lump to identify it.

ZDoom and Eternity already had a lot of parser code that could be easily co-opted for UDMF. We don't have any of that so I'd have to write it all. Never mind all that weird stuff with line tags GZ mentioned that makes no sense to me (I suppose you need to know the differences between Doom and Hexen's map formats, but I know nothing about Hexen and don't care to.)

Basically you're going from a patch that touches a few dozen places in obvious ways to likely hundreds and hundreds of lines of new code which due to my utter incompetence would inevitably be full of hilarious bugs.

Anyway enough derailing this thread, sorry kickassand.

RjY said:

Yes but I am thoroughly unmotivated to write patches for support of even basic Doom-namespace UDMF just for limit removal in PrBoom, when I know all that it really needs is for a handful of the fields in the map structs to go from 16 to 32 bits. And to shove a magic number on the start of the lump to identify it.

But even if you did something like that you'd be on your own because you lack the most crucial feature of all: Editor support!

RjY said:

Basically you're going from a patch that touches a few dozen places in obvious ways to likely hundreds and hundreds of lines of new code which due to my utter incompetence would inevitably be full of hilarious bugs.

Anyway enough derailing this thread, sorry kickassand.

That's one reason I originally argued for separate line tag, for DOOM-based namespaces, and ID/arg0 for Hexen, because I knew this would come up. Implementation is impeded by anything that's perceived as a widespread need for change in the engine.

However, I don't think it's as complex as you're probably thinking.

To implement support for the DOOM namespace, all you actually need to do is expect to read tags either from arg0 or id. This is really just an input issue. An implementing port which does not have Hexen support should just end up putting either id or arg0 into its internal line_t::tag field. The spec states that for DOOM, Heretic, and Strife maps, the id and arg0 will be equal.

As for skill levels, UDMF's base spec allows specification of all 5 skill levels separately. This just requires a trivial allowance in P_SpawnMapThing, so I don't think it's a big deal. You could even neglect to support combinations of skill levels that you don't support, but it would make your implementation not strictly compliant, and would create compatibility problems with other ports that do implement the separated skill levels if people chose to use that feature.

Maps converted to the namespace from the binary format already wouldn't include "invalid" combinations of skill levels; the only concern is that they might be added during subsequent editing. It was my original intent that UDMF maps under compatibility namespaces remain strictly backwards-compatible except with respect to numerical limits, but this was overruled by other parties.

Quasar said:

That's one reason I originally argued for separate line tag, for DOOM-based namespaces, and ID/arg0 for Hexen, because I knew this would come up. Implementation is impeded by anything that's perceived as a widespread need for change in the engine.

However, I don't think it's as complex as you're probably thinking.

To implement support for the DOOM namespace, all you actually need to do is expect to read tags either from arg0 or id. This is really just an input issue. An implementing port which does not have Hexen support should just end up putting either id or arg0 into its internal line_t::tag field. The spec states that for DOOM, Heretic, and Strife maps, the id and arg0 will be equal.

Uh, no. That won't work.
For Doom, Heretic and Strife binary formats both are equal. What you suggest here will pretty much guarantee that some maps won't work if they use both values. Remember the endless discussions we had about this particular issue when we discussed UDMF?

Anyway, separating these 2 values is, maybe 20-30 minutes of work if you know the code, so what's the fuss about? (I'd be more than happy to help should it ever come up.)

Graf Zahl said:

Uh, no. That won't work.
For Doom, Heretic and Strife binary formats both are equal. What you suggest here will pretty much guarantee that some maps won't work if they use both values. Remember the endless discussions we had about this particular issue when we discussed UDMF?

Anyway, separating these 2 values is, maybe 20-30 minutes of work if you know the code, so what's the fuss about? (I'd be more than happy to help should it ever come up.)

Oh, because again the maps might be subsequently edited. Right.

For maps converted TO the compatibility namespaces, what I said is true. But if they're edited it might not be. Blah. :P

Precisely.

Graf Zahl, not at all. I would certainly be in favor of using an existing widespread format as a foundation to build off of. I was merely expressing some concerns I thought important. It's quite possible that a significant portion of developers/users do not share the same concerns, and would be glad to accept as-is ZDoom's format as standard; it was my intention for a thread like this to help to find out.

RjY, no worries. With a standard texture format on so many back-burners, perhaps we can afford another UDMF rehashing. BTW, your situation regarding implementing a UDMF parser from scratch is one of the the reasons I think an up-to-date, tested reference implementation is a good thing to have when trying to push a standard.

The reason there's no reference implementation for UDMF is that the three driving developers for the specs were working with C, C++ and C#. Three different languages. (I don't remember a move to C++ having been planned at the time for Eternity. If I'm not mistaken, this jump was prompted by silliness from GCC with regards to type casts and disappointment with the evolution of the C standard; not with UDMF's requirements.)

To top it off, the ZDoom implementation was based on the SC parser from Hexen, while the Eternity implementation would be based on the libconfuse parser. (I don't know about the DB2 implementation.)

There was no real point in creating a reference implementation in this case. Plus, ZDoom and DB2 are open-source, if you want reference implementations. Instead, the standard was written so that it wouldn't be hard to implement it with whatever existing architecture you already have available. SLADE3's implementation of UDMF (very limited right now since there's no map editor yet) was written using Simon Judd's custom parser which he originally wrote for parsing config files and for handling syntax highlighting in text lumps.

Pr/GLBoom+ already has parsers -- it can handle its config files, it can handle DEHACKED lumps with both the DeHackEd and the BEX syntaxes (and these are two completely different syntaxes and they can be mixed within the same file, so the parser must necessarily handle more complicated scripts than a UDMF map), it can handle the MUSINFO lump, and it can handle GLDEFS lumps. The same parsing code could be used to parse UDMF maps.

Gez, thank you again for the relevant info. I have to say, while I was not necessarily looking to it initially, the more I research the more XML seems a natural lifeboat in this ocean of text formats (the number of which I hadn't realized was so large, and the syntaxes so varied). Many birds could be killed with that stone: description, testing, verification, integration, adaptation..

I understand size issues related to the format, but .NET's System.Xml is nice, and both libxml++ and pugixml are adequate for quick drop-in solutions (as well as uncountable numbers of other APIs sporting a variety of licensing choices). Certainly a homegrown parser need not be a requirement for using XML?

XML had been considered and rejected during the debate threads on UDMF. You can find them linked to from here.

It depends on what you want to do.

XML is good for complex structures that are entirely machine generated and maintained. So for a map format it might have been ok but since all other text lumps are supposed to be hand-editable XML is not really an option.

But for a map format it would have had one major problem which pretty much ruled it out and that is its insane verboseness. Large UDMF maps already approach 15 MB of text. Imagine that with a format that insists on repeating each keyword and you end up with twice that. At some point it just gets too large.

Plus, we wanted something that's easy to code (not the parser but the evaluator for the parsed data) and doing that with XML is a lot more complex than a simple 'key = value' format which, if you already got a working parser can be done in a handful of lines of code.

If you take a look at ZDoom's p_udmf.cpp you'll see that less than 10% of the entire file are actual parsing code. Most of it are large switch/case blocks that assign the parsed values to the proper variables and all of this would have had to be there anyway, regardless of the format the map is stored in.

Gez, I have seen those threads recently and noticed the issues some had with size (both of the format and of the libraries used to parse it) and ease of (human) use. I deal with those issues as well, though as Graf Zahl mentions, the latter can be mostly academic for certain applications. Regarding the size of the format specifically, do many existing source ports/editors already support compressed archives?

In any case, it would seem that a standard text format would benefit the community. I haven't really seen a text format that is specially suited to certain data or that couldn't be replaced with another (theoretically, mind you. I understand some code bases are more tightly-coupled in this area than others). If I understand the landscape: there are a handful of source ports that have various proprietary formats, and also generally prefer to support the others' proprietary formats as well. This seems like quite a bit of unnecessary work. Is this an accurate assessment of the situation?

Graf Zahl said:

Of course, if there's a general interest I wouldn't stand in the way but seeing how UDMF went with only ZDoom implementing it (and Vavoom copying ZDoom's implementation) I think it's obvious where this is heading.

From a map editors point of view, I found UDMF to be a big "win" because I felt like I had to jump through fewer hoops to get ZDoom effects to do what I wanted. Despite the lack of uptake in other ports, I'm glad you, Quasar and CodeImp went through the trouble of drafting a new map format.

As such, even if port X had implemented feature A long before port Y. Port Y will often duplicate feature A's functionality in an incompatible way, effectively creating a feature Ab, A2 etc.

kickassand said:

Gez, I have seen those threads recently and noticed the issues some had with size (both of the format and of the libraries used to parse it) and ease of (human) use. I deal with those issues as well, though as Graf Zahl mentions, the latter can be mostly academic for certain applications. Regarding the size of the format specifically, do many existing source ports/editors already support compressed archives?

In any case, it would seem that a standard text format would benefit the community. I haven't really seen a text format that is specially suited to certain data or that couldn't be replaced with another (theoretically, mind you. I understand some code bases are more tightly-coupled in this area than others). If I understand the landscape: there are a handful of source ports that have various proprietary formats, and also generally prefer to support the others' proprietary formats as well. This seems like quite a bit of unnecessary work. Is this an accurate assessment of the situation?

One thing that's important to consider is that no matter how well you can compress the data for disk storage, you still have to parse all of it to actually load the map in your source port. The more unnecessarily bloated the map format is, the more data there is to read into memory when loading a sufficiently sized level. That in itself is a pretty good motivation to avoid something like XML (as was suggested during UDMF's development) for Doom levels, which not unfrequently have multiple tens of thousands of entries making up their map geometry. Each of those entries also having all of their attributes specified XML-style is going to be absolutely insane.

UDMF is already a bit big for comfort.

esselfortium, thanks for the reply.

Regarding compression, I was suggesting it might be practical to allow (in whatever standard specs may be created/modified) these text lumps to be stored in compressed form, using some quick, easy and prevalent method (such as deflate). One can then decompress and parse on-the-fly (if one wishes to reduce the memory load). zlib of course supports in-memory streaming, and C++ users can opt to use Boost.Iostreams to work with zlib compressed data as any other std:: stream. (There are a myriad of other more lightweight solutions, the advantage of choosing an established algorithm such as deflate.)

As you say, text representations of these data are going to be large whatever the format, so such an allowance in a standard spec does not seem inappropriate to me.

Why add another layer of complexity? ZDoom already supports Zips as container format but so far I haven't seen the need to add a parser that can read directly from a zlib stream. I'm not bothered by allocating 15-20 MB to parse a map on systems that have 30-200 times as much RAM.

When writing a spec the most important thing is: Keep it simple! Any needless comlplexity, and even compression is that, will hinder its adoption because it means that potential implementors first have to implement other things they don't really need.

The same thing has pretty much made it impossible to turn ZDoom's compressed nodes into a standard. Only after I added an uncompressed variant of them things moved forward. I think that's saying all about your latest ideas.