Trying to determine the purpose of apparently unused bytes in IWADs

Aha! I've got it - thanks for your post @fraggle, it set me down the right road.

It turns out that they are in fact padding bytes and, instead of the (I would think) obvious choice of padding with null bytes, the end of each lump is padded with the first byte in the lump. The DEMO1 example in the OP: The DEMO1 lump data is padded with 0x6D bytes because the first byte of the DEMO1 lump is 0x6D.

We have the source code to some of the Doom utilities but I don't believe we have the source to wadlink, the tool id used to build WADs. So we have to kind of infer.

The 4 byte boundary thing makes sense - I know some x86 CPUs process data less efficiently when it's not aligned. I didn't realise that included the 486 but I'm not too surprised. Some CPU architectures straight up do not support reading unaligned data and will fault.

However it's a bit weird. What I've just said relates to memory alignment, not alignment of data on disk. One answer might be that it was added in a console port (maybe the Jaguar port Carmack did himself?) - in that case the WAD is just part of the ROM. One thing to check would be if the padding was missing in the early IWADs and appeared in later versions. Maybe they were working on the Jaguar Port and just happened to be using the same tool to build both versions.

As for the data itself, it's probably random data from wadlink's memory at the time it was run. Maybe the same buffer used to write the lumps themselves - see if you can find the same byte sequence in the preceding lump.

EDIT: I guess I was confused because we do have the source code to wadlink: check here for the code that ensures the alignment. Since it writes a byte at a time the padding will always be the same byte, but it does reuse the buffer. But no comments to hint as to why it's aligned in the first place.

EDIT 2: Looks like versions of Doom before v1.2 didn't do the 4 byte alignment thing. So it was probably added early in 1994 (v1.2 was released February 17th).

Note that's around when both 32X and Jag Doom were being worked on, and that in both, the entire IWAD is mapped to memory as part of the ROM, and therefore is subject to those systems' rather stringent data alignment requirements. I get the distinct feeling that it is in fact related.