Using wadlc and wadldc in the build

[selliott4]

The recent the "Wadspy output with Daily Builds" thread having to do with summarizing levels with a text file made me curious if it's possible to take the concept one step further where the level WADs could be entirely converted to and from text format. The "wadlc" and "wadldc" CLIs from xwadtools, the same GitHub Doom-Utils organization Freedoom gets its "deutex" from, are such tools.

 

This would make it possible for the actual checked in Git resource in to be a text file, so something like "levels/map11.txt" instead of "levels/map11.wad". The build would then produce all of the "levels/*.wad" files based on the "levels/*.txt" files. There would be some other build target, maybe "submit", to go the other way to update "levels/*.txt" files based on modified "levels/*.wad" files. This would be done after editing "levels/*.wad" files.

 

Why the additional abstraction? The levels would then be diffable and mergeable. When viewing pull requests for levels you'd be able to actually see a summary of  the changes in GitHub, although cryptic. More importantly if ongoing work from multiple authors made changes to different parts of the level they could be merged by Git automatically. Diffs could be done locally as well to make sure no unintended changes were made before submitting a pull request.

 

What do you guys think? Maybe I could look into modifying the build if there is any interest.

[Voros]

I don't see how this is convenient for people that aren't familiar with GitHub. Doom WADs are made using simple yet powerful WAD editors like Slade3, GZDoom Builder, etc. It's how Doom content was made to be and shared among other with ease. 

 

WADs are easily editable with all the editors that have a GUI these days. People don't want to make WADs from a text file (maybe for shits and giggles, because I'm interested in checking this strange thing out).

 

So no, not really. The "Use Wadspy to output level stats" thing is a useful thing that should be implemented by fraggle on the device (Raspberry Pi IIRC) that handles autobuilds. This looks like an obscure way to use ancient software to build WADs, when people just make WADs with a WAD editor.

 

[selliott4]

Thanks for the feedback, but just to clarify - I was assuming that people would continue using the same WAD editors they are currently using, that part of the process would be the same. I wasn't assuming that anyone would want to update WADs by editing the text files. Only the build would handle the conversion either direction.

 

As to people not familiar with GitHub - my post was geared toward that because I assumed that you want people to do use that process, but indeed all the current ways need to be accommodated:

1. People doing the full GitHub pull request approach. They would just do "make submit" after their normal process when done with their edits to the levels, but before the pull request.
2. People with commit access pushing directly to GitHub.  They would just do "make submit" after their normal process when done with their edits to the levels, but before the git push.
3. People who have the build setup, but don't use GitHub and who currently post their WADs somewhere on the internet. They would just do "make submit" after their normal process and then post the text files somewhere on the internet instead of the WAD files.
4. People who don't have the build setup, but edit the levels anyway. This is a trickier. They'll have to get the build working, or work with wadlc and wadldc directly.

That's my understanding of what people are doing.

 

EDIT: Possibly for steps 3 and 4 the submitter would just submit the WAD files and the maintainer would put them in a build tree and do "make submit" before committing the changes.

 

I'd totally understand if this seems too odd, but I thought I'd suggest it in case it's helpful.

 

I agree that "Use Wadspy to output level stats" thread is a good idea regardless.

 

By the way, do you have some way of comparing to versions of a level WAD graphically, like something that displays the changes made on a 2D image, so you can see what was done? I was assuming that was not common when I posted this. I guess that's another topic, but I have some ideas on that as well.

Edited March 21, 2017 by selliott4

[Voros]

Another thing: when people posts a new map for Freedoom, it's ALWAYS a WAD. Freedoom uses Deutex to compile all the necessary files. For map files, it has to be WAD (else Deutex won't read it), which it is always.

 

If Freedoom does use that software, all your doing is adding some unnecessary steps to the build process. The simpler the process is, the better.

 

And no, no such program exists that can do that. There is a person here who was made several Java programs that users asked for. You could look into requesting him into asking if it's possible for a small program.

 

But I think Slade/Doom Builder would be better, as they always have BAK files for WADs, so the program could have a seperate function that records the layout of the map in the current WAD as a image file (for example) and do the same for the latest BAK file, and then, possibly put them side by side in a whole a new image file. Doesn't sound that easy, but my knowledge of programming is too basic right now, but it's not impossible.

 

Or for convenience, run a function that compares the technical details of the current map, store the data somewhere. Then making some edits, resave the map and run the function and store this new data along side the older one. Another function will take the latest data and the data before it and place them on the view (kinda like building the map layout, but not he actual map), possibly showing stuff like changed linedefs and things as red things while non chnaged stuff as blue. This idea is very very basic and needs more planning out to do, but I hope you get what I mean.

Edited March 21, 2017 by Voros

[RjY]

On 21/03/2017 at 1:04 PM, selliott4 said:

Why the additional abstraction? The levels would then be diffable and mergeable.

For a diff what you want is a small change by the person editing giving a small difference to the resulting file. Unfortunately with wads a small change made in the map editor will likely spew changes all over the file, due to the way the map structs all refer to each other by array index. Whether the map is in binary wad format, or wadldc or likewise output, doesn't make much difference.

 

If the commit only adjusts linedef/sidedef/sector properties without renumbering anything you can get a fairly sensible diff with a simple text dump of the map struct arrays but as soon as the person editing moves anything around, copies/deletes an object, or rebuilds nodes, the diff becomes unworkably large.

 

________

 

I've had some success with diffing wads using git's textconv filters. Try hooking wadldc into git as one of these. It will need a small wrapper script since wadldc needs to be told both the name of the map to dump (even if there's only one in the file), and to write to stdout explicitly, to turn it into a filter. The following worked for me (assuming wadldc and lswad are on your $PATH)

#!/bin/sh

while [ $# -gt 0 ] ; do
  for map in `lswad $1 | sed -n '/THINGS/ { g;p }; h'` ; do
    wadldc $map $1 /dev/stdout
  done
  shift
done

(Bah, no syntax highlighting for shell scripts.)

Anyway, then hook it into textconv. Tell your freedoom tree about wad files, then configure wad files to use the script as a textconv driver:

cd freedoom
echo '*.wad diff=wad' >> .git/info/attributes
git config diff.wad.textconv <PATH_TO_ABOVE_SCRIPT>

You should then be able to see stuff like this:

% git show master
commit 85dd14b34d760835831913a60ac19a7b7f406bc0
Author: Catoptromancy <catoptromancy@yahoo.com>
Date:   2017-03-17 19:30:04 -0700

    c4m9: Easy skill rebalance, added soulsphere and extra shells.

diff --git a/levels/c4m9.wad b/levels/c4m9.wad
index 58e8458d..e728a426 100644
--- a/levels/c4m9.wad
+++ b/levels/c4m9.wad
@@ -15886,7 +15886,7 @@ LEVEL_START 4 9 0 Doom2
   9 : 4256 288 270 6
   3001 : 4320 288 270 7
   3001 : 4384 288 270 6
-  58 : 4384 224 270 7
+  58 : 4384 224 270 6
   3004 : 4384 160 270 23
   3002 : 4256 160 270 4
   9 : 4256 224 270 7
@@ -16037,7 +16037,7 @@ LEVEL_START 4 9 0 Doom2
   2012 : -2436 212 315 15
   3001 : -1984 616 315 14
   3001 : -2072 952 315 15
-  2012 : 2400 352 315 15
+  2012 : 2404 352 315 14
   2048 : 2272 -32 315 15
   2028 : 1472 1088 315 15
   2035 : 1032 264 270 23
@@ -16862,6 +16862,17 @@ LEVEL_START 4 9 0 Doom2
   2011 : 2816 292 315 1
   2011 : -1436 -308 315 3
   27 : -236 -1028 270 7
+  2013 : 2396 352 0 1
+  2007 : 1984 240 0 1
+  2007 : 1952 240 0 1
+  2008 : 1584 96 0 1
+  2008 : 1584 64 0 1
+  2008 : 80 -1312 0 1
+  2008 : -1264 -1072 0 1
+  2008 : -592 -1040 0 1
+  2008 : -496 -1040 0 1
+  2048 : -816 -1456 0 1
+  2008 : -848 -2528 0 1
  THINGS_END

 LEVEL_END

See Performing text diffs of binary files in man:gitattributes(5).

[selliott4]

Good stuff. Thanks for sharing that. With regard to diffing the IDs getting renumbered what if they IDs were stripped out before the diff?  Of course you then have less information. If only each item had some ID that never got changed once set.

 

Also, this probably means that my original goal of having maps be git mergeable won't work in most cases :-/

[Ferk]

I think this looks really cool from a developer's point of view, but very awkward from a mapper's point of view.

 

Probably a project like this should target making life easier for the artists/mappers, who are the people Freedoom should try to attract, since they are the ones contributing the content.

I don't think having to convert the txt files all the time will be very comfortable. It looks like some people even have trouble with making clean git commits, this wouldn't improve it.

 

Maybe if the editing tools were able to use this txt format directly in some way, or there were some very convenient scripts that could be run in all platforms (yes, including Windows) that were not a pain to set up with multiple dependencies.

And even then, many mappers might still prefer to just make a wad, since it's what everyone is used to do already.

[chungy]

RjY's git hackery doesn't require changes to the repo contents itself, it can still store binary WAD files for levels. I think it's the best route to go if somebody wants this information out of git.

[RjY]

19 hours ago, selliott4 said:

With regard to diffing the IDs getting renumbered what if they IDs were stripped out before the diff?  Of course you then have less information.

This works. But one would also like to preserve the relationships between objects, by listing those referenced alongside their parents. I conjectured two formats:

• list each linedef, the position of each of its vertexes, the properties of its one or two sidedefs, and the sector properties on either side, as a single record. This can work really well or it can completely break locality when a single sector property is changed (all the sector's lines will be detected as changed, as opposed to a simple listing of sectors separately).
• list each sector, followed by each line surrounding it (basically doing the equivalent of P_GroupLines) then changing a linedef property will change only two places at most. (Hopefully. I haven't tried writing this yet.)

And yes the loss of information is very unfortunate. You see something has changed in a textconv diff, you go to look at it in a graphical map editor, but the map editor wants an object number to jump to, which you don't have!

 

19 hours ago, selliott4 said:

If only each item had some ID that never got changed once set.

Yes, quite. Curse Carmack's lack of foresight! :) Perhaps there is some way to hash each object which preserves locality, but that's beyond me.

 

19 hours ago, selliott4 said:

Also, this probably means that my original goal of having maps be git mergeable won't work in most cases :-/

Indeed this seems to be as far-off a goal as ever. I expect nothing but a full on graphical tool for merge resolution would be adequate. Like vimdiff, but part of a map editor. Somebody patch this into eureka...

 

 

11 hours ago, chungy said:

I think it's the best route to go if somebody wants this information out of git.

Agreed -- barring some major unforeseen change in map editing technology, I expect the binary wad format to remain the "preferred form of modification" for maps, as it has been for the past 20+ years.