New tool/suite: hwadtools (v0.5.0: lswad, wadxtract, waddiff, wadcat, wadstats) source & binaries for linux/macos/windows

Edit: README for the suite is here

Last night I wrote (in a kind of speedmapping style) a tool "lswad", the first of what will perhaps be a suite of simple tools in the UNIX philosophy for manipulating Doom WADs etc., I'm calling "hwadtools". Inspired somewhat by xwadtools, but a bit more modern (xwadtools seems to be a PITA to build or use on modern systems).

I'm mostly doing this as an exercise in writing Haskell programs.

"lswad" is a simple tool that lists the contents of WAD files.

Example:

$ ./lswad jonhex.wad
  name  	  size  	 offset
MAP01   	0       	12
THINGS  	10      	12
LINEDEFS	47950   	24
SIDEDEFS	205080  	47976
VERTEXES	10472   	253056
SEGS    	83328   	263528
SSECTORS	11140   	346856
NODES   	77952   	357996
SECTORS 	30264   	435948
REJECT  	169362  	466212
BLOCKMAP	16128   	635576
GL_MAP01	92      	651704
GL_VERT 	2852    	651796
GL_SEGS 	107540  	654648
GL_SSECT	11140   	762188
GL_NODES	77952   	773328
GL_PVS  	0       	851280
WADCSRC 	3293    	851280

It's actually inspired by a similar tool I wrote in C perhaps 10 years ago, which was probably inspired by xwadtools at the time. I found it handy enough that I was still using it recently.

Source code and binaries for Windows (64-bit), Linux (64-bit) and OS-X are here:

https://github.com/jmtd/hwadtools/releases/tag/v0.5.0

Like I said I've approached this a bit like speed mapping, I could gild the lily a lot more but I decided to get this out as soon as it seemed functionally complete. There are no doubt bugs and a few style warts.

Bug, questions, comments, suggestions for other tools to add to the suite, etc. are all welcome.

Finally I aim to collect/collate a related project "badwads", containing carefully constructed problem WADs to expose bugs in WAD-handling tools (things ranging from invalid magic to directory offset past the end of file, and suchlike). If you know of any or have any such WADs already please let me know!

On my possibility list: deutex (except reproducible such that wad -> explode -> implode === wad for any wad); a wad compressor; reimplement inkfoundry; a wad normaliser/de-linter;  node builder. 

Pushed v0.2.0 source tonight; binaries tomorrow. 

New is a (very rough) "wadxtract" tool.

edit: binaries up https://github.com/jmtd/hwadtools/releases/tag/v0.2.0

Dusted this off after a break -- I'd spent a lot of time on a "wadbuild" dual of "wadxtract" but they (both) need a lot more work.

So I did something completely different and wrote a preliminary "waddiff". Here's an example of a WAD before and after running ZDBSP

$ waddiff jonhex.wad jonhex2.wad
--- jonhex.wad
+++ jonhex2.wad
@@
 MAP01    (0 bytes)
 THINGS   (10 bytes)
-LINEDEFS (47950 bytes)
+LINEDEFS (47950 bytes)
@@
 SIDEDEFS (205080 bytes)
-VERTEXES (9048 bytes)
+VERTEXES (9360 bytes)
+SEGS     (83136 bytes)
+SSECTORS (11024 bytes)
+NODES    (77140 bytes)
@@
 SECTORS  (30264 bytes)
+REJECT   (0 bytes)
+BLOCKMAP (16124 bytes)
@@
 WADCSRC  (3275 bytes)

Source and binaries coming soon

Very cool stuff. Careful on the offset checks. I assume you're already thinking about this: a very tricky program could compress a WAD with maybe some success, by manipulating offsets in a way that a lump could point to another lump's data.

It's an interesting idea, and probably has already been done. But I fear that some WAD editing tools might very well flip out on such WADs, corrupting them on edit/save. In fact, all of these tools should be identifying such WAD directories, and auto-expand them, at least in memory, before trying to allow manipulation of the WAD. My guess is that very few, if any editors consider this possibility. And, as long as the referenced lumps are not edited, and edits are confined to areas past such data, editors may very well save these lumps in such WADs properly, by just writing out those directory entries as-is.

Really what I'm trying to say is that WAD compression via manipulation of the directory is dangerous - like a hidden land mine in the WAD.

I'm not sure, but it's possible that id Software's WAD tools might have been able to do partial WAD edits by saving new versions of lumps at the end of the file, and updating the directory entry, leaving unused space where the old lump existed. Back then, rewriting the whole WAD took a long time, but doing one of these edits would have been much quicker. Modern WAD editors would generally handle this, and, depending on how they are written, would probably clean up this dead space on save. But in the case of lump entries pointing to other lumps? Sketchy. A WAD editor needs to actively know about this, to handle it properly.

What do you think?

By the way, nice tools!

OK v0.3.0 incorporating waddiff is now available. Source, and mac, linux and windows binaries

https://github.com/jmtd/hwadtools/releases/tag/v0.3.0

16 hours ago, kb1 said:

Very cool stuff. Careful on the offset checks. I assume you're already thinking about this: a very tricky program could compress a WAD with maybe some success, by manipulating offsets in a way that a lump could point to another lump's data.

Yes for wadbuild/wadxtract I want them to be able to deconstruct and reconstruct any WAD that might exist. They already handle holes but handling overlapping resources is something they should do too. The other tools are read-only at the moment and work fine with such things (I believe)

I think Fraggle's wadptr compressor can do some of those things . Various source ports will not accept the resulting WADs though, e.g. prboom uses mmap (I believe) to map PWADs into memory and makes layout assumptions that break down.

V0.4.0 released: https://github.com/jmtd/hwadtools/releases/tag/v0.4.0

This released adds a "wadcat" tool. I wasn't sure whether to write this one, I was pondering its utility since it doesn't do any rewriting (so if you cat two MAP01 PWADs together, you get two MAP01s). Then I saw two things: xwadtools had one, and I stumbled across a random PWAD on /idgames that mentioned it had used the xwadtools wadcat to build it. Also I had in the back of my mind it could be useful for something else I was planning, so I wrote it.

Love that you're doing this -- command line tools for DOOM formats are seriously lacking making automation very difficulty.

I have a project called DOOM-Crusher that unpacks WADs/PK3s, compresses the resources inside (jpgtran/pngout etc.) and then repacks the file. Currently I use a modified version of "lumpmod", an old and creaky tool that lists the contents of WAD files. In order to automate compression, a friendly doomer modified it to include an extra column that gives the file-type of the lump, i.e. JPG, PNG so that I know what lumps to extract without having to extract and examine all of them.

If you could add this feature, with a broader set of recognised file-types, that would be helpful to everybody working with WADs.

I have another project, PortaDOOM, which is both a launcher and '90s style disk-zine WAD collection. It contains a 1500+ line batch file that can take WAD files and a desired engine and it will launch the engine with all the necessary and correct command-line switches as well as managing config files, save files (separating them by IWAD/PWAD) and numerous other behaviours designed to make a massive variety of engines all behave the 'same'. All of this has taken a great deal of time to build up and good quality DOOM command-line tools make it easier.

At this moment I'm working on expanding my launcher to more intelligently manage the highly complex relationship between WADs <> engines <> MODs.

• Different engines support different features (e.g. Boom, UDMF, DECORATE etc.)
• WADs might include different amounts of modifying content, e.g. DEH, ACS, ZScript
• This affects what mods would be compatible, based on what the mod also modifies (monsters, weapons etc.)

How this is all managed at the moment is that I work out this compatibility and encode it in some meta-data (command-line params / .ini files) so that PortaDOOM can pass on what a WAD requires and my launcher can find the compatible engines accordingly. This limits my launcher to PortaDOOM's curated list of WADs.

What the DOOM world is missing is better tools to test and identify compatibility.

Is there any way you can build a tool that can find what features a WAD is using?

• Are Boom, MBF, SMMU etc. features being used?
• Is ACS present?
• Is DECORATE present?
• Is ZScript present, what functions are being called? (these would narrow down the exact GZDoom version needed)
• Is UDMF being used?
• Is this using extended (>32K) nodes? GL nodes?

Information about these features is scattered to the winds with no specific details readily available on which engines support what. This can be worked out over time, but we need to begin with the ability to even tell what is being used in a WAD/PK3 file as a guidance to compatibility.

Thanks for the comments! Your projects sounds really interesting.

9 minutes ago, Kroc said:

If you could add this feature [file type guessing/sniffing], with a broader set of recognised file-types, that would be helpful to everybody working with WADs.

That's certainly something I can consider. It's related to another idea I was thinking about which was to have a switch for all the tools to output in a structured format for machine consumption. Something like --json for example.

9 minutes ago, Kroc said:

Is there any way you can build a tool that can find what features a WAD is using?

• Are Boom, MBF, SMMU etc. features being used?
• Is ACS present?
• Is DECORATE present?
• Is ZScript present, what functions are being called? (these would narrow down the exact GZDoom version needed)
• Is UDMF being used?
• Is this using extended (>32K) nodes? GL nodes?

Information about these features is scattered to the winds with no specific details readily available on which engines support what. This can be worked out over time, but we need to begin with the ability to even tell what is being used in a WAD/PK3 file as a guidance to compatibility.

That's a great idea. A big piece of work but a valuable tool it would be. I'll put it on my list.

I know it's a big ask. It's the kind of thing that needs to start small and build over time because the community as a whole needs to be slowly steered into the right direction of taking compatibility seriously. Look at the HTML/CSS world where there's caniuse.com which can tell you exactly what browsers/versions can use what features.

I'd be happy to help with such a tool. I can't program C (only web-languages and Golang, but I might learn Rust), though I can help with testing, information gathering and advanced batch scripting. My launcher can already automate over 30+ DOOM engines/versions.

The current implementation of wadcat (in the latest release, v0.4.0) writes out to the last argument you supply to it. so, running "wadcat a.wad b.wad c.wad" would concatenate a.wad and b.wad into the output file c.wad.

I'm planning to change this in the next release to be more in line with the UNIX cat tool. Both for consistency with that, and to reduce the risk of a user accidentally overwriting one of their existing WADs. Instead, all command-line arguments will be input WADs, and the output will be written to standard output. However, since nobody ever wants to display a PWAD's constituent binary in a command prompt, you will be required to redirect it (or wadcat will report an error instead). E.g. "wadcat a.wad b.wad > c.wad" will work. "wadcat a.wad b.wad" will complain that you haven't redirected the output.

I've just released v0.5.0. This adds a new tool "wadstats" which prints out statistical info about the input PWADs, in YAML. Example in spoiler tags

$ ./wadstats unreal8.wad 
- MAP01:
    vertices: 2056
    sectors: 443
    linedefs: 1769
    things:
      count: 40
      Singleplayer:
       "player 1 start": 1
       "player 2 start": 1
       "player 3 start": 1
       "player 4 start": 1
       "11": 6
       "14": 4
       "cellchargepack": 1
       "doublebarreled": 1
       "85": 1
       "shotgun": 3
       "chaingun": 3
       "rocketlauncher": 2
       "plasmagun": 1
       "shotgunshells": 4
       "medikit": 4
       "greenarmor": 1
       "boxofrockets": 2
       "cellcharge": 1
       "boxofammo": 2

A bit of a delay since the last release; I wrote 90% of wadstats months ago, but I couldn't decide for myself on the output format, so I shelved it for a while until I had a flash of inspiration and came up with what we have here.

Source and binaries for Linux and OS X: https://github.com/jmtd/hwadtools/releases/tag/v0.5.0

One caveat: I did the Windows builds last night and I've since lost them somehow, so they are not available until I find them or redo them, possibly tonight.

@Kroc I've re-read your suggestions and I have some follow-up comments finally

On 2/15/2018 at 1:18 PM, Kroc said:

I have a project called DOOM-Crusher that unpacks WADs/PK3s, compresses the resources inside (jpgtran/pngout etc.) and then repacks the file. Currently I use a modified version of "lumpmod", an old and creaky tool that lists the contents of WAD files. In order to automate compression, a friendly doomer modified it to include an extra column that gives the file-type of the lump, i.e. JPG, PNG so that I know what lumps to extract without having to extract and examine all of them.

Hm. I'm not sure about adding this to lswad. It means re-implementing something like file(1), since a lump could be anything really. However if I had a truly idempotent wadxtract/wadbuild, so you could safely explode/implode any WAD without fear of breaking anything, one could build some scripting around that and file(1) perhaps.

On 2/15/2018 at 1:18 PM, Kroc said:

Is there any way you can build a tool that can find what features a WAD is using?

• Are Boom, MBF, SMMU etc. features being used?
• Is ACS present?
• Is DECORATE present?
• Is ZScript present, what functions are being called? (these would narrow down the exact GZDoom version needed)
• Is UDMF being used?
• Is this using extended (>32K) nodes? GL nodes?

Information about these features is scattered to the winds with no specific details readily available on which engines support what. This can be worked out over time, but we need to begin with the ability to even tell what is being used in a WAD/PK3 file as a guidance to compatibility.

These are fantastic ideas and I'm considering them for wadstats.

Last spam, I found the windows EXEs so the windows release has been attached at the github page. Enjoy!