New tool/suite: hwadtools (v0.4.0: lswad, wadxtract, waddiff, wadcat) source & binaries for linux/macos/windows

Last night I wrote (in a kind of speedmapping style) a tool "lswad", the first of what will perhaps be a suite of simple tools in the UNIX philosophy for manipulating Doom WADs etc., I'm calling "hwadtools". Inspired somewhat by xwadtools, but a bit more modern (xwadtools seems to be a PITA to build or use on modern systems).

 

I'm mostly doing this as an exercise in writing Haskell programs.


"lswad" is a simple tool that lists the contents of WAD files.

 

Example:

Spoiler

$ ./lswad jonhex.wad
  name  	  size  	 offset
MAP01   	0       	12
THINGS  	10      	12
LINEDEFS	47950   	24
SIDEDEFS	205080  	47976
VERTEXES	10472   	253056
SEGS    	83328   	263528
SSECTORS	11140   	346856
NODES   	77952   	357996
SECTORS 	30264   	435948
REJECT  	169362  	466212
BLOCKMAP	16128   	635576
GL_MAP01	92      	651704
GL_VERT 	2852    	651796
GL_SEGS 	107540  	654648
GL_SSECT	11140   	762188
GL_NODES	77952   	773328
GL_PVS  	0       	851280
WADCSRC 	3293    	851280

 

It's actually inspired by a similar tool I wrote in C perhaps 10 years ago, which was probably inspired by xwadtools at the time. I found it handy enough that I was still using it recently.

 

Source code and binaries for Windows (64-bit), Linux (64-bit) and OS-X are here:

https://github.com/jmtd/hwadtools/releases/tag/v0.1.0

 

Like I said I've approached this a bit like speed mapping, I could gild the lily a lot more but I decided to get this out as soon as it seemed functionally complete. There are no doubt bugs and a few style warts.

 

Bug, questions, comments, suggestions for other tools to add to the suite, etc. are all welcome.

 

Finally I aim to collect/collate a related project "badwads", containing carefully constructed problem WADs to expose bugs in WAD-handling tools (things ranging from invalid magic to directory offset past the end of file, and suchlike). If you know of any or have any such WADs already please let me know!

Edited by Jon
3 people like this

Share this post


Link to post

On my possibility list: deutex (except reproducible such that wad -> explode -> implode === wad for any wad); a wad compressor; reimplement inkfoundry; a wad normaliser/de-linter;  node builder. 

Share this post


Link to post

Dusted this off after a break -- I'd spent a lot of time on a "wadbuild" dual of "wadxtract" but they (both) need a lot more work.


So I did something completely different and wrote a preliminary "waddiff". Here's an example of a WAD before and after running ZDBSP

$ waddiff jonhex.wad jonhex2.wad
--- jonhex.wad
+++ jonhex2.wad
@@
 MAP01    (0 bytes)
 THINGS   (10 bytes)
-LINEDEFS (47950 bytes)
+LINEDEFS (47950 bytes)
@@
 SIDEDEFS (205080 bytes)
-VERTEXES (9048 bytes)
+VERTEXES (9360 bytes)
+SEGS     (83136 bytes)
+SSECTORS (11024 bytes)
+NODES    (77140 bytes)
@@
 SECTORS  (30264 bytes)
+REJECT   (0 bytes)
+BLOCKMAP (16124 bytes)
@@
 WADCSRC  (3275 bytes)

Source and binaries coming soon

2 people like this

Share this post


Link to post

Very cool stuff. Careful on the offset checks. I assume you're already thinking about this: a very tricky program could compress a WAD with maybe some success, by manipulating offsets in a way that a lump could point to another lump's data.

 

It's an interesting idea, and probably has already been done. But I fear that some WAD editing tools might very well flip out on such WADs, corrupting them on edit/save. In fact, all of these tools should be identifying such WAD directories, and auto-expand them, at least in memory, before trying to allow manipulation of the WAD. My guess is that very few, if any editors consider this possibility. And, as long as the referenced lumps are not edited, and edits are confined to areas past such data, editors may very well save these lumps in such WADs properly, by just writing out those directory entries as-is.

 

Really what I'm trying to say is that WAD compression via manipulation of the directory is dangerous - like a hidden land mine in the WAD.

 

I'm not sure, but it's possible that id Software's WAD tools might have been able to do partial WAD edits by saving new versions of lumps at the end of the file, and updating the directory entry, leaving unused space where the old lump existed. Back then, rewriting the whole WAD took a long time, but doing one of these edits would have been much quicker. Modern WAD editors would generally handle this, and, depending on how they are written, would probably clean up this dead space on save. But in the case of lump entries pointing to other lumps? Sketchy. A WAD editor needs to actively know about this, to handle it properly.

 

What do you think?

 

By the way, nice tools!

Share this post


Link to post

OK v0.3.0 incorporating waddiff is now available. Source, and mac, linux and windows binaries

 

https://github.com/jmtd/hwadtools/releases/tag/v0.3.0

 

 

16 hours ago, kb1 said:

Very cool stuff. Careful on the offset checks. I assume you're already thinking about this: a very tricky program could compress a WAD with maybe some success, by manipulating offsets in a way that a lump could point to another lump's data.

 

Yes for wadbuild/wadxtract I want them to be able to deconstruct and reconstruct any WAD that might exist. They already handle holes but handling overlapping resources is something they should do too. The other tools are read-only at the moment and work fine with such things (I believe)

 

I think Fraggle's wadptr compressor can do some of those things . Various source ports will not accept the resulting WADs though, e.g. prboom uses mmap (I believe) to map PWADs into memory and makes layout assumptions that break down.

 

Share this post


Link to post

V0.4.0 released: https://github.com/jmtd/hwadtools/releases/tag/v0.4.0

 

This released adds a "wadcat" tool. I wasn't sure whether to write this one, I was pondering its utility since it doesn't do any rewriting (so if you cat two MAP01 PWADs together, you get two MAP01s). Then I saw two things: xwadtools had one, and I stumbled across a random PWAD on /idgames that mentioned it had used the xwadtools wadcat to build it. Also I had in the back of my mind it could be useful for something else I was planning, so I wrote it.

Share this post


Link to post

Love that you're doing this -- command line tools for DOOM formats are seriously lacking making automation very difficulty.

 

I have a project called DOOM-Crusher that unpacks WADs/PK3s, compresses the resources inside (jpgtran/pngout etc.) and then repacks the file. Currently I use a modified version of "lumpmod", an old and creaky tool that lists the contents of WAD files. In order to automate compression, a friendly doomer modified it to include an extra column that gives the file-type of the lump, i.e. JPG, PNG so that I know what lumps to extract without having to extract and examine all of them.

 

If you could add this feature, with a broader set of recognised file-types, that would be helpful to everybody working with WADs.

 

I have another project, PortaDOOM, which is both a launcher and '90s style disk-zine WAD collection. It contains a 1500+ line batch file that can take WAD files and a desired engine and it will launch the engine with all the necessary and correct command-line switches as well as managing config files, save files (separating them by IWAD/PWAD) and numerous other behaviours designed to make a massive variety of engines all behave the 'same'. All of this has taken a great deal of time to build up and good quality DOOM command-line tools make it easier.

 

At this moment I'm working on expanding my launcher to more intelligently manage the highly complex relationship between WADs <> engines <> MODs.

 

portadoom_launcher.png.bcca8f8d37bcfb4c279a125c93292aef.png

 

  • Different engines support different features (e.g. Boom, UDMF, DECORATE etc.)
  • WADs might include different amounts of modifying content, e.g. DEH, ACS, ZScript
  • This affects what mods would be compatible, based on what the mod also modifies (monsters, weapons etc.)

How this is all managed at the moment is that I work out this compatibility and encode it in some meta-data (command-line params / .ini files) so that PortaDOOM can pass on what a WAD requires and my launcher can find the compatible engines accordingly. This limits my launcher to PortaDOOM's curated list of WADs.

 

What the DOOM world is missing is better tools to test and identify compatibility.

 

Is there any way you can build a tool that can find what features a WAD is using?

  • Are Boom, MBF, SMMU etc. features being used?
  • Is ACS present?
  • Is DECORATE present?
  • Is ZScript present, what functions are being called? (these would narrow down the exact GZDoom version needed)
  • Is UDMF being used?
  • Is this using extended (>32K) nodes? GL nodes?

Information about these features is scattered to the winds with no specific details readily available on which engines support what. This can be worked out over time, but we need to begin with the ability to even tell what is being used in a WAD/PK3 file as a guidance to compatibility.

1 person likes this

Share this post


Link to post

Thanks for the comments! Your projects sounds really interesting.

 

9 minutes ago, Kroc said:

If you could add this feature [file type guessing/sniffing], with a broader set of recognised file-types, that would be helpful to everybody working with WADs.

 

That's certainly something I can consider. It's related to another idea I was thinking about which was to have a switch for all the tools to output in a structured format for machine consumption. Something like --json for example.

 

9 minutes ago, Kroc said:

Is there any way you can build a tool that can find what features a WAD is using?

  • Are Boom, MBF, SMMU etc. features being used?
  • Is ACS present?
  • Is DECORATE present?
  • Is ZScript present, what functions are being called? (these would narrow down the exact GZDoom version needed)
  • Is UDMF being used?
  • Is this using extended (>32K) nodes? GL nodes?

Information about these features is scattered to the winds with no specific details readily available on which engines support what. This can be worked out over time, but we need to begin with the ability to even tell what is being used in a WAD/PK3 file as a guidance to compatibility.

 

That's a great idea. A big piece of work but a valuable tool it would be. I'll put it on my list.

Share this post


Link to post

I know it's a big ask. It's the kind of thing that needs to start small and build over time because the community as a whole needs to be slowly steered into the right direction of taking compatibility seriously. Look at the HTML/CSS world where there's caniuse.com which can tell you exactly what browsers/versions can use what features.

 

I'd be happy to help with such a tool. I can't program C (only web-languages and Golang, but I might learn Rust), though I can help with testing, information gathering and advanced batch scripting. My launcher can already automate over 30+ DOOM engines/versions.

1 person likes this

Share this post


Link to post

The current implementation of wadcat (in the latest release, v0.4.0) writes out to the last argument you supply to it. so, running "wadcat a.wad b.wad c.wad" would concatenate a.wad and b.wad into the output file c.wad.

 

I'm planning to change this in the next release to be more in line with the UNIX cat tool. Both for consistency with that, and to reduce the risk of a user accidentally overwriting one of their existing WADs. Instead, all command-line arguments will be input WADs, and the output will be written to standard output. However, since nobody ever wants to display a PWAD's constituent binary in a command prompt, you will be required to redirect it (or wadcat will report an error instead). E.g. "wadcat a.wad b.wad > c.wad" will work. "wadcat a.wad b.wad" will complain that you haven't redirected the output.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now