Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Sign in to follow this  
pivotal

WAD File Structure

Recommended Posts

I'm not sure if this is the correct forum for this, so a mod can move it if there's a better sub-forum for this.

I am writing a command-line launcher for doom source ports in python. I'd like to be able to associate PWADs with the correct IWAD. I've figured out how to determine whether a WAD is an IWAD or PWAD by reading the first 4 bytes, but I'd like to take it a step further. I'd like the user to be able to select a PWAD and have it automatically choose the correct IWAD.

I've done a bit of googling for a document that outlines the structure of a WAD file, but I haven't turned up anything all that great. So, my question is this: does anyone have a document that describes the structure of Doom WAD files?

Once I get my program working reasonably well, I'll throw it up on github if anyone is interested.

Share this post


Link to post
pivotal said:

I'm not sure if this is the correct forum for this, so a mod can move it if there's a better sub-forum for this.

I am writing a command-line launcher for doom source ports in python. I'd like to be able to associate PWADs with the correct IWAD. I've figured out how to determine whether a WAD is an IWAD or PWAD by reading the first 4 bytes, but I'd like to take it a step further. I'd like the user to be able to select a PWAD and have it automatically choose the correct IWAD.

I've done a bit of googling for a document that outlines the structure of a WAD file, but I haven't turned up anything all that great. So, my question is this: does anyone have a document that describes the structure of Doom WAD files?

Once I get my program working reasonably well, I'll throw it up on github if anyone is interested.


http://doom.wikia.com/wiki/WAD

Good place to start with the basic format. This document also has a lot both about the basic format as well as individual lumps:

http://www.gamers.org/dhs/helpdocs/dmsp1666.html

As far as autodetect, good luck on that. You might start by saying things like "well, if it's got ExMy, it's Doom; if it has MAPxy, it's Doom2", but what about Heretic? And what about source ports that support non-standard map lump names? So for the former, you start scumming through the maps for texture names or thing types, and for the latter, you start parsing things like ZDoom's MAPINFO lump. To differ between doom2 and tnt/plutonia, you have to get into texture stuff as well.


I'm sure you could make a basic iwad detector rather easily, but you could spend quite a long time on edge cases and whatnot.

Share this post


Link to post

Edit: ninjaed by the less verbose Natt who pretty much made exactly all the same points as I did; just faster.

pivotal said:

I am writing a command-line launcher for doom source ports in python. I'd like to be able to associate PWADs with the correct IWAD.

The best you can do is some guesswork. Making something 100% reliable would be exceedingly complicated.

pivotal said:

I've done a bit of googling for a document that outlines the structure of a WAD file, but I haven't turned up anything all that great. So, my question is this: does anyone have a document that describes the structure of Doom WAD files?

The structure of a WAD file is extremely simple. Header points to directory, directory points to each lump.
Here's a full summary.

What you can try is search for map lumps. If they're in the form ExMy, you can assume they're for Ultimate Doom; if they're in the form MAPxx, you can assume they're for Doom II.

However, there are plenty of other IWADs. Maybe it require TNT.wad or Plutonia.wad rather than doom2.wad. Detecting such cases from a simple scanning of the directory would be impossible; you'd have to analyze the map lumps -- notably the SIDEDEFS lumps -- to find references to TNT- or Plutonia-specific textures. Which also requires making lists of each IWAD's textures. You'd also have to scan the WAD's TEXTURE1 and TEXTURE2 lumps, if any, to check whether they haven't imported textures from another IWAD -- a very common practice which could induce false positives.

And it becomes complicated because there are WADs for other games than Doom; even if there aren't as many. Heretic and Chex Quest use ExMy maps too; Hexen and Strife use MAPzz.

The issue becomes even further muddled when you have ports, like ZDoom, that can use any arbitrary name for a map thanks to MAPINFO lumps. For example, ZPack uses the ExMy naming convention but it's for Doom II anyway. However, ZDoom mods tend to be in PK3 (aka ZIP) archives rather than WAD anyway.

Note that none of this is directly related to the WAD format.


The thing to keep in mind is just that unless you start keeping databases in your application, you'll have a system that might work most of the times (the overwhelming majority of PWADs ever made is for Doom II, followed by Ultimate Doom; everything else is a tiny minority) but still commit a mistake from time to time; so keep a way for the user to override automatic detection.

Share this post


Link to post

A better idea might be to have your launcher work on the zip file straight from idgames. Have it extract the info text and try to figure out the correct port/IWAD from that. It's impossible for it to work in all cases, but there's a ton of levels that are just text file + WAD.

Share this post


Link to post
david_a said:

A better idea might be to have your launcher work on the zip file straight from idgames. Have it extract the info text and try to figure out the correct port/IWAD from that. It's impossible for it to work in all cases, but there's a ton of levels that are just text file + WAD.


Heh reminds me of all those "social network" web services that try to infer context automatically by scraping blogs/tweets/facebook wall posts.

Share this post


Link to post
david_a said:

A better idea might be to have your launcher work on the zip file straight from idgames. Have it extract the info text and try to figure out the correct port/IWAD from that. It's impossible for it to work in all cases, but there's a ton of levels that are just text file + WAD.

The directory structure in /idgames distinguishes between the IWADs (sort of). I.e. there's /idgames/levels/doom, /idgames/levels/doom2, /idgames/levels/heretic etc. So fullsort.gz could be used, if the file is in the archive.

Share this post


Link to post
boris said:

The directory structure in /idgames distinguishes between the IWADs (sort of).


However, there are a few mistakes.

Share this post


Link to post
Gez said:

The best you can do is some guesswork. Making something 100% reliable would be exceedingly complicated.

Well, if it's not ZDoom, Eternity or any other advanced port, you can guess whether it's Doom or Doom 2 by scanning for ExMy or MAPxy. Further, you can guess if it requires one of the Final Dooms by scanning for required textures. If it still WAS designed either for Doom 2 or Final Doom, you can either prompt the player the way ZDoom does, scan the accompanying TXT file for information or whatever you wish...

Share this post


Link to post

Everything they said, plus more.

The tests will be an unbounded, you might as well plan an expanding the tests on a regular basis, as new discrepancies are discovered, or new ports with new features arise.

Also need to determine the correct advanced port:
DoomLegacy has some unique sidedef type codes.
ZDoom has some unique sidedef type codes.

There is some IWAD wad recognition code in ZDoom (and in DoomLegacy) that already does a good job of determining the IWAD. I recommend looking at the GDESC_ code in DoomLegacy d_main.c as it is table based.
The same texture names appearing in a PWAD will restrict the possible IWADS.

There have been a few times I wished I had a tool to determine this, as many PWAD do not work as the notes described.

Some wads require more help.
Running caesar.wad on DoomLegacy, using FreeDoom IWAD requires also loading gothic2.wad, to get the PNAMES lump. Using the Doom original IWAD does not have this problem. Yet, if you are using FreeDoom because you do not have the original Doom wads, then it is rather important.

Download XWADTOOLS (for Linux), which has many little programs that already does some of this work, and has some docs.

You might start by deliberately running PWAD with the wrong IWAD, and seeing how the port errors occur. Then code a similar usage as a test (probably can get a good start from the port code that detects the error).

Share this post


Link to post

Hey hey hey these are not enterprisey solutions. Clearly, we must define an XML-based metadata standard for WAD files, to be embedded as a lump in WAD files. Did I mention it would have to be XML-based?

Share this post


Link to post
Maes said:

Hey hey hey these are not enterprisey solutions. Clearly, we must define an XML-based metadata standard for WAD files, to be embedded as a lump in WAD files. Did I mention it would have to be XML-based?


Dude, plain old XML is sooo last century man! You might as well be writing in COBOL! If it ain't AJAX with a bunch of half-baked frameworks, then it ain't Web 2.0 compliant! I'd also recommend ditching those ancient scripting languages and bone up on some OMG Ruby on Railz! :P

Seriously now, I like the CLI front-end idea. One feature that you might also consider is the ability to pull ratings/reviews from the doomworld database. It should be really trivial, if you can obtain the id <-> file mapping they use in their database. For example, TeamTNT's Icarus megawad is id 5191:

http://www.doomworld.com/idgames/index.php?id=5191

And you can easily pull just the reviews:
http://www.doomworld.com/idgames/iframe_review.php?id=5191

There might even be a way to grab the rating score(s) also, without scraping the index.php page.

You could go further still, and allow submitting ratings/comments, since it's only necessary to solve a simple math problem. Then again, the site admins may or may not like that, but it's something to think about...

Share this post


Link to post

Wow, I didn't expect this much feedback! I went out of town shortly after I posted the thread, and I was without internet access, so I haven't responded yet.

natt said:

To differ between doom2 and tnt/plutonia, you have to get into texture stuff as well.
I'm sure you could make a basic iwad detector rather easily, but you could spend quite a long time on edge cases and whatnot.


I figured that making something that works 100% of the time would be difficult, so I'm going to shoot for 95%+, and let the user correct any mistakes. I hadn't thought about checking texture names, that's a good idea.

Gez said:

What you can try is search for map lumps. If they're in the form ExMy, you can assume they're for Ultimate Doom; if they're in the form MAPxx, you can assume they're for Doom II.

However, there are plenty of other IWADs. Maybe it require TNT.wad or Plutonia.wad rather than doom2.wad. Detecting such cases from a simple scanning of the directory would be impossible; you'd have to analyze the map lumps -- notably the SIDEDEFS lumps -- to find references to TNT- or Plutonia-specific textures.

The thing to keep in mind is just that unless you start keeping databases in your application, you'll have a system that might work most of the times (the overwhelming majority of PWADs ever made is for Doom II, followed by Ultimate Doom; everything else is a tiny minority) but still commit a mistake from time to time; so keep a way for the user to override automatic detection.


I figured out that I can search for ExMy and MAPxx to narrow down the search, but that still leaves the problem of PWADs that use TNT/Plutonia/Hexen/Strife/Chex Quest.

Like I said above, I'm not shooting for 100% accuracy. You make a good point about overriding autodetect. I figure I could try matching, and then display to the user what IWAD matched, and if it is incorrect, let the user specify the correct IWAD.

Also, to get around having to include a database of all known PWADs, I've been experimenting with searching the idgames database and scraping info from there. If a PWAD runs down the list of tests to determine the IWAD, and a conclusive match isn't found, I could scrape the info from the idgames database to get the correct IWAD.

wesleyjohnson said:

Everything they said, plus more.

Also need to determine the correct advanced port:
DoomLegacy has some unique sidedef type codes.
ZDoom has some unique sidedef type codes.

There is some IWAD wad recognition code in ZDoom (and in DoomLegacy) that already does a good job of determining the IWAD. I recommend looking at the GDESC_ code in DoomLegacy d_main.c as it is table based.
The same texture names appearing in a PWAD will restrict the possible IWADS.

There have been a few times I wished I had a tool to determine this, as many PWAD do not work as the notes described.

Some wads require more help.
Running caesar.wad on DoomLegacy, using FreeDoom IWAD requires also loading gothic2.wad, to get the PNAMES lump. Using the Doom original IWAD does not have this problem. Yet, if you are using FreeDoom because you do not have the original Doom wads, then it is rather important.

Download XWADTOOLS (for Linux), which has many little programs that already does some of this work, and has some docs.

You might start by deliberately running PWAD with the wrong IWAD, and seeing how the port errors occur. Then code a similar usage as a test (probably can get a good start from the port code that detects the error).


Thanks, this is all useful. Looking at code someone else wrote to solve a similar problem is always helpful. I'll check out XWADTOOLS to learn more.

Maes said:

Hey hey hey these are not enterprisey solutions. Clearly, we must define an XML-based metadata standard for WAD files, to be embedded as a lump in WAD files. Did I mention it would have to be XML-based?


Haha, if I went down this path, I could use this hardware XML appliance from IBM:
http://www-01.ibm.com/software/integration/datapower/xi50/features/

hex11 said:

Seriously now, I like the CLI front-end idea. One feature that you might also consider is the ability to pull ratings/reviews from the doomworld database. It should be really trivial, if you can obtain the id <-> file mapping they use in their database. For example, TeamTNT's Icarus megawad is id 5191:

http://www.doomworld.com/idgames/index.php?id=5191

And you can easily pull just the reviews:
http://www.doomworld.com/idgames/iframe_review.php?id=5191

There might even be a way to grab the rating score(s) also, without scraping the index.php page.

You could go further still, and allow submitting ratings/comments, since it's only necessary to solve a simple math problem. Then again, the site admins may or may not like that, but it's something to think about...


After I read the first couple posts in this thread, I started playing around with scraping idgames for info instead of trying to detect a bunch of difficult edge cases. Then I realized I could easily get ratings, allow WAD downloads, etc. Submitting ratings and reviews is a good idea, I'll consider that. I'll get in touch with whoever runs idgames first to get their approval.

Thanks for all the suggestions so far. I'll try to keep this thread updated.

Oh, one more idea I had was to do user-defined aliases. Let's say you're recording speedruns, and you're sick of typing out the command with arguments every time. My goal is to let you pick all the arguments you want to use, and save it as an alias so you can do it quickly.

Well, time to get coding.

Share this post


Link to post
Maes said:

Hey hey hey these are not enterprisey solutions. Clearly, we must define an XML-based metadata standard for WAD files, to be embedded as a lump in WAD files. Did I mention it would have to be XML-based?

We use Info files with addons for Doomsday/Doomsday frontend.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  
×