WadSmoosh - retail WAD merge tool for ZDoom

Figured it was time for WadSmoosh to have its own thread here. The original release has been on idgames for almost a year but I've since done many small fixes and improvements that add up to a lot, so I'm looking to verify that the current dev build is rock solid then upload an update to idgames.

Main project page

Source code
Screenshot:

What does WadSmoosh do? It merges your official (released by id) PC retail IWADs into a single PK3 IWAD for ZDoom called "doom_complete.pk3" that contains (if you own all the official releases) Ultimate Doom, Doom 2 and its two addons, and Final Doom. GZDoom 2.4 and later recognize the WadSmoosh IWAD automatically.

What doesn't WadSmoosh do? Merge any non-official, console and/or freely available content. Please don't ask me to add support for smooshing Lost Episodes, PlayStation or Doom64 TC, etc. See this post in another thread for my thoughts on the need for an easy-to-use arbitrary WAD merging tool - tl;dr WadSmoosh was built to do a single thing well and I'm not going to overhaul it into SLADE-lite.

I'm also not interested in adding support for other source ports - a lot of what makes WadSmoosh possible is ZDoom-specific - unless I deem the changes required trivial.

WadSmoosh's Python source code is included with every build so you can modify it as you wish. Feel free to fork it. Bug reports and pull requests for fixes are welcome!

The testing I'd like to do before calling the current dev build good enough for a 2.0 update on idgames is mainly just making sure it provides the same behavior as if you were running with Doom.wad / Doom2.wad - I want users to be able to use this as their daily driver IWAD and not worry about compat. There are a couple tough exceptions I can't fix cleanly though:

• Final Doom contains texture definitions that conflict with Doom2's, so I renamed those and replace them at runtime with ACS ZScript. This means that the few mods that require TNT.wad or Plutonia.wad may not look exactly as they do with those original IWADs - Plutonia 2 for example doesn't show the correct sky, but nothing major breaks.
• Doom 2 PWADs that redefine SKY1/2/3 in a certain way (via the TEXTURE1 lump) won't show the right sky. As of this week WadSmoosh has an options screen section where you can enable something called "legacy sky support" which should fix this up. For the vast majority of WADs you can leave this off. Wish this could have been auto-detected in ACS or ZScript somehow but them's the breaks.

This looks pretty cool and works well from the VCS HEAD on my Arch system :)

Might I suggest enabling compression for the pk3 members though? This can bring the file size in towards the size of doom2.wad ;)

diff --git a/wadsmoosh.py b/wadsmoosh.py
index f6e1e4e..941b9d8 100644
--- a/wadsmoosh.py
+++ b/wadsmoosh.py
@@ -1,7 +1,7 @@
 
 import os, sys
 from shutil import copyfile
-from zipfile import ZipFile
+from zipfile import ZipFile, ZIP_LZMA
 
 import omg
 
@@ -316,7 +316,7 @@ def main():
         add_xbox_levels()
     # create pk3
     logg('Creating %s...' % DEST_FILENAME)
-    pk3 = ZipFile(DEST_FILENAME, 'w')
+    pk3 = ZipFile(DEST_FILENAME, 'w', ZIP_LZMA)
     for dir_name, x, filenames in os.walk(DEST_DIR):
         for filename in filenames:
             src_name = dir_name + '/' + filename

18 hours ago, chungy said:

This looks pretty cool and works well from the VCS HEAD on my Arch system :)

 

Might I suggest enabling compression for the pk3 members though? This can bring the file size in towards the size of doom2.wad ;)

 

That would make it a PK7, I suppose? I avoided using Python LZMA because it's not installed by default on macOS and some Linux distros, and my goal for those users was to make sure the program can be run from the included source script using whatever Python is built in to the OS (usually Python 2.7).

There's also a bit more of a launch speed penalty when booting GZDoom and loading levels with a PK7 IWAD, but it's not a huge issue.

PK7 is normally the 7z file format, which only has a benefit for solid compression (which is actually not good for a game engine to load random files from the archive, that's what makes it slow :P). PK3 is Zip, and using LZMA, we can get the same compression as a non-solid 7z.

All Linux distros from the past 7 years or so include LZMA by default. I never use Mac OS X, so I can't speak for it, but maybe something like this would work:

from zipfile import ZipFile
try:
    from zipfile import ZIP_LZMA as ZIP_COMPRESS
except:
    from zipfile import ZIP_DEFLATED as ZIP_COMPRESS

# some lines down to the ZipFile() method...
pk3 = ZipFile(DEST_FILENAME, 'w', ZIP_COMPRESS)

DEFLATE is supported by practically everything that supports Zip (eg: the Windows Explorer zip support doesn't support anything but stored and deflated archive members). It'll yield a bit worse compression, but it should still be much better than storing everything uncompressed.

The main problem with LZMA is that some Zip tools cannot do anything but Deflate, the most annoying being WxWidgets' internal Zip decompressor. And we got one editor that's using this framework...

Just confirmed ZIP_DEFLATED works with OS X's shipped Python, and ZIP_LZMA doesn't.

On 2017-07-01 at 3:30 PM, JPL said:

There's also a bit more of a launch speed penalty when booting GZDoom and loading levels with a PK7 IWAD, but it's not a huge issue.

I disagree. I think this is a huge penalty, and it is not worth the extra compression especially since the resulting .pk3 is meant for local storage and not distribution. I think it would be better to use regular deflate compression, at the very least by default if an option must be offered for LZMA. It also would make GZDoom consume more resources since it has to both decompress and also keep the entire structure inside RAM, and with the systems that GZDoom is typically run on this can present a problem especially with larger mods.

That being said, the pk3 should definitely be Deflate compressed by default, in my opinion.

On 2017-07-02 at 2:40 AM, Graf Zahl said:

The main problem with LZMA is that some Zip tools cannot do anything but Deflate, the most annoying being WxWidgets' internal Zip decompressor. And we got one editor that's using this framework...

 

The one that's become pretty much the defacto standard for wad editing, these days, considering it's the only active project and also is open source (which apparently, its predecessors were originally not).

If I had more time on my hands I would just drop in a decent compression library and plug it in, and submit a pull request.

Thanks for the discussion folks, I'll try out Deflate and commit once I've tested on Win/Mac/Lin.

Okay, new build is up with deflate compression (and a new ENDOOM with better credits):

https://jp.itch.io/wadsmoosh

Also for those interested in doing a 162-level run through every possible map in the complete WadSmoosh IWAD, see the "Doom Complete Playthrough" mod I just posted on my page: https://jp.itch.io/doom-fullrun