Interesting idea -- XWAD

I had a small flash of brilliance. The existing WAD format is identified by the IWAD or PWAD identifier, which is the first byte of the file. The header is a minimum of 12 bytes, but can be padded out to any length since the WAD format, like any appropriate archive format, is completely based on offsets and not on absolute file positions.

Taking these together, you could have the XWAD format with support for long lump names. Here's how it might look:

First byte of header: XWAD
0x0c-0x0f: XWAD format version
0x10-0x13: Offset to XWAD directory

Note that the normal directory can still exist in the normal header data. This means that with the small adjustment of 1. accepting XWAD in the first byte and 2. not stomping on the additional two header words, existing WAD utilities could still load XWADs and use them as normal wads (an external utility would be needed to update the XWAD directory if any changes were made, however, and that might get complicated...)

XWAD directory structure (per-record):
0x00-0x03: offset (unsigned long)
0x04-0x07: lump size (unsigned long)
0x08-0x4b: lump name (64 bytes, null extended)

With this, you could easily give WADs a pseudo-directory structure, since the file name is long enough to contain classifications. You could also give lumps extensions to better indicate the format of the data inside them.

An example name possible:
eternity/heretic/font/fonta93.patch

In only 35 chars I've established that this is a lump belonging to the source port, that it's the heretic version of it, it's part of the font, and that it is in DOOM's patch graphics format. Wow, neato.

This could enable stuff like multiple versions of the same lump in a wad and the ability for ports to load them without repeated checks -- it could set the "basepath" for those resources to, for example, "eternity/heretic" or "eternity/doom" and automatically have every correct version of the lump load by simply using the constructed lump name:

snprintf(buf, sizeof(buf), "%s/%s", basepath, lumpname);

Let's see if this discussion devolves as quickly as my 32-bit level format idea :P

Sounds like a great concept; as a matter of fact, I was thinking the other day that it's a pity that Boom didn't start something like this back in '98. It would have sorted out compatibility issues and encouraged development, perhaps (although arguably now with Doom+ in some cases that line gets blurry.)

I like the idea. Though this could be a pain with compatibility issues.

Doomsday already supports something very similar with the DD_DIREC lump which was implemented a fair number of years back. It basically allows for any lumps in the WAD to be organised into a virtual directory structure and also allows data lumps to be marked up as file types (eg a music lump can be specified directly as being MUS, MIDI, MP3 etc).

Skyjake even wrote a utility (wadtool available from doomsdayhq.com) that would take any folder/directory structure, write the DD_DIREC lump and pack the files into the WAD automatically.

For whatever reason, the feature was never widely used outside of the dev team and the Korax mod though.

Once PK3 support was implemented - there was nolonger a reason to develop DD_DIREC further and 99% of Doomsday mods now use PK3.

This sounds very useful, and perhaps more compatible than altering the WAD format itself. Are there any specs or code available for it? I'm not beyond coming up with my own implementation, but there's no need to reinvent the wheel in this case if there's something really good already out there :)

Inventing a new WAD format seems like a waste of effort considering that you might as easily use something more standardized like Zip. You already got tons of tools that support it - as well as some source ports so Wad manager support isn't completely unlikely.

Graf Zahl said:

Inventing a new WAD format seems like a waste of effort considering that you might as easily use something more standardized like Zip. You already got tons of tools that support it - as well as some source ports so Wad manager support isn't completely unlikely.

Only if the beauty and elegance of an extended format that could live inside the old one is completely lost on you. But that aside, if this were any other game engine I would agree -- zip is a good format for games, and in fact my generic C++ SDL game engine framework uses zip files through the PhysicsFS library.

I do not see zips being a particularly good fit for DOOM, however. They do not fold into the existing code at all.

I can see already, however, that a virtual directory structure is a better fit than a real one built into the wad format itself. I am intrigued about the idea of a 2nd level hash table that maps directory paths to lumps. Eternity needs something like this to avoid a massive code rewrite and addition of dozens of annoying branches that would otherwise be necessary in order to fold eternity.wad and eterhtic.wad together. Once Hexen support work begins (which is really not that far off), the problem will really become severe if I do not solve it now.

Graf Zahl said:

Inventing a new WAD format seems like a waste of effort considering that you might as easily use something more standardized like Zip.

I take it you remember the nice big discussion about it over on the ZDoom forums a few years back? :)

Having not kept track of ZDoom, I have no idea whether ZIP support was added, but alot of time discussing here would be better spent reading that thread as you'll find alot of the same for/against points will come up.

Quasar said:

I do not see zips being a particularly good fit for DOOM, however. They do not fold into the existing code at all.

Why not? If done right you don't really need to change anything beyond w_wad.c. In ZDoom I only had a problem with music because it was streamed directly out of the WAD but even that change cost me only a few hours. All the rest was done internally to the WAD manager and the rest of the code never knows where the data comes from. It just continues to use the same functions it used before without any change.

The only non-straightforward thing I did was to map some directories into the global namespace to keep things a little more manageable.

But you do get the advantage of having full paths if needed. This was my prime motivation to add it in the first place - to support models in GZDoom.

GooberMan said:

I take it you remember the nice big discussion about it over on the ZDoom forums a few years back? :)

Having not kept track of ZDoom, I have no idea whether ZIP support was added, but alot of time discussing here would be better spent reading that thread as you'll find alot of the same for/against points will come up.

Yes, I added Zip-support and if the DRDteam website was accessible to me right now I could point you to the specifications. If done properly you can get a nice mapping of directories to WAD namespaces. The one issue I didn't address was splitting up maps into their lumps. ZDoom's zip support only allows to place the entire map as a WAD into it at the moment.

And yes, I remember that thread - including the most vocal (wasn't he the only one? ;)) opponent and his mostly nonsensical arguments.

This sounds very useful, and perhaps more compatible than altering the WAD format itself. Are there any specs or code available for it? I'm not beyond coming up with my own implementation, but there's no need to reinvent the wheel in this case if there's something really good already out there :)

There are no written specs to my knowledge but the source code is available for both the wadtool utility and the code required to intergrate support for DD_DIREC in Doomsday itself.

However I must warn you that it is completely intergrated with Doomsday's virtual file system (which intergrates lumps and 'real' data files loaded from WAD, PK3 and hard files, so isolating the code for DD_DIREC might be tricky (eg a seperate DD_DIREC.lmp can be loaded via the command line which can then reorganise data files loaded from PWADs, IWADs the /auto folders or even PK3s, before then being hashed into the 'real' virtual file system)).

It's somewhat complex but insanely flexible though, as such it was well worth the considerable time skyjake spent on designing it.

I BELIVE that the EDGE team where working on a wad format where you could create your "wad" by arranging bitmaps, ddf's, sounds and the like into folders in windows, and then use a 'zip' program to pack it into a wad-like format for use

I'm gonna speak up for the zip format here as well. While its groovy that you can actually extend the wad format, I can't really see any reason why you'd want to keep using it. In itself it doesn't really have any notable qualities, aside from simplicity. So if you're going to make a new format, why can it not be something that doesn't even require any specialised tools?

Theres only a few problems that arise from this. One is that most wad editors handle other tasks as well, such as sprite offsets, texture definitions etc. Aside from that, all they do is get in the way. So the first step is to negate the need for that altogether. My favourite solution would be for these functions to be extended onto the current crop of map editors, although command line tools or text format definitions based on the old formats would do the job. Most people would rather add new textures using the Zdoom method anyway I'd wager.

Another is that after all that work, you don't really see much benefit from using zip instead of wad. You still need to compile the lumps into a single file to make it work. Well its high time that all doom ports supported loading data straight from a directory tree, thus eliminating the need for a container altogether until you actually want to upload something. If you built computers as a job, you wouldnt climb inside the box while you were working on one, would you?

Here is my two cents. For EDGE we started developing a wad extension system, the details can be seen here:

http://edge.sourceforge.net/wad_sys_II.txt

The big problem with any new format is tool support. It's all very nice to add this into the engine, but users need a way of using it. Do you really want to create a new packaging tool? I didn't (especially since there are no small open-source cross-platform packaging tools around to use as a base).

My suggestion is to use ZIP files, develop a heirarchy standard, and rewrite w_wad.c to load normal WADs into that standard. The above document may give you some ideas.

chilvence said:

Well its high time that all doom ports supported loading data straight from a directory tree, thus eliminating the need for a container altogether until you actually want to upload something. If you built computers as a job, you wouldnt climb inside the box while you were working on one, would you?

No professionally made commercial game engine that I'm aware of keeps all its resources in separate files. There are many good reasons for this, and I wouldn't think of dumping the contents of eternity.wad out into a directory tree. It's fine for files and directories to be able to be mapped into a virtual file system (which things like PhysicsFS seamlessly allow), but to distribute data in that format is disadvantageous.

No professionally made game packages its resources in seperate files on the disc or when its installed on your hard drive, but a fair few allow for it for one good reason - its a hell of a lot easier to work with if you're any kind of developer.

So you have misread my argument, I wasn't suggesting you dump the contents of any wad on a users hard drive as seperate files, but you should certainly be able to work with seperate files as most wad editors dont understand what half of them are and take aeons to add token support for new ones. Whats the point of working from inside a wad if most of the tools you use aren't even related to doom?

Ajapted said:

http://edge.sourceforge.net/wad_sys_II.txt

IMO that's way too messy and unintuitive.
Especially concerning levels you may want to read this.
The big problem is that if you really split up the levels you make yourself utterly dependent on new tools. For other types of data it's much simpler. All you'd need is extend XWE or any other lump manager to write Zips. XWE already can read them but logically that's only half the job.
But if you keep the levels intact you can simply extract them with any Zip management tool available and use any existing tool you like with them.

Since I had to add Zip support to GZDoom in order zto support models properly I faced the same problem. However, I wanted it to keep as simple as possible. Here is how I defined it. It's not 100% complete because I still have to separate the special directories from the global namespace but if you keep the restrictions coming from that in mind it works quite well.

My suggestion is to use ZIP files, develop a heirarchy standard, and rewrite w_wad.c to load normal WADs into that standard. The above document may give you some ideas.

That's exactly how I did it.

Graf Zahl said:

All you'd need is extend XWE

you can't extend something EASILY if the source isn't available for it.

unless you have 1337 binary editing skillz

RTC_Marine said:

you can't extend something EASILY if the source isn't available for it.

Too bad, isn't it? :(

Graf Zahl said:

IMO that's way too messy and unintuitive.

Probably. I'm not going to defend a unused spec I wrote two years ago too vigorously, but one requirement was being able to load both DOOM and HERETIC iwads (for example) at the same time and play their levels with the correct textures/flats/etc. Can GZDOOM do that?

Ajapted said:

Probably. I'm not going to defend a unused spec I wrote two years ago too vigorously, but one requirement was being able to load both DOOM and HERETIC iwads (for example) at the same time and play their levels with the correct textures/flats/etc. Can GZDOOM do that?

No. But that's not due to the WAD organization. The WAD manager is flexible enough to selectively find a lump in specific WADs or other resource files.

But there's just too many parts in the game that access the data in a way that makes using duplicate names problematic. For example the level management code has been designed with the assumption that each map name exists only once so mixing Doom.wad and Heretic.wad will make one set of levels inaccessible unless it was rewritten to accomodate this. Other problems are global resources (like the PLAYPAL and COLORMAP.) Again, not a problem of the file management but the rendering code simply wouldn't know that there were multiple instances for those and which one to use. However, if there was any need whatsoever to add this it could be done - but what's the point? ;)

For us the point was making the engine not so hard-coded to Doomisms.