Doom Standard Idea: Demo Metadata

[RestlessRodent]

I had this idea (spoken of at https://www.doomworld.com/vb/post/1424451) where LMPs could have attached metadata in the form of a ZIP file after the end of a demo.

EDIT: I removed most of the implementation details in this post, it can instead just be read at the linked document instead.

The current specification which I have defined is at http://remood.org:8080/remood/doc/ckout/other/ddmd.txt.

Basically it allows for an easy extendable enchancment of demos by adding extra meta-data to end of the demo file. This would allow for utilities and source ports to better cooperate and work with demo files.

[fraggle]

Why a ZIP file? Sounds like overkill.

IIRC PrBoom(+) already does this. It would make more sense to just standardize what it's already doing than invent a new "standard".

[RestlessRodent]

fraggle said:

Why a ZIP file? Sounds like overkill.

• Better support with libraries and utilities compared to WADs. Java, .NET, and Python come with ZIP support in their standard library. Other languages have libraries for managing reading/writing ZIP files. One could also write their own ZIP loading code and use miniz if desired.
  
• Compression.
  
• Directories.
  
• Filenames above 8 characters.
  
• Central index is at the end of the file, rather than requiring a header at the start.
  
• The ZIP can contain a file which is the actual demo content (if ever desired).
  

fraggle said:

IIRC PrBoom(+) already does this. It would make more sense to just standardize what it's already doing than invent a new "standard".

PrBoom(+) uses a WAD. The WAD location is based off the location of the first 0x80 found at a specific alighnment in the file (based on the size of each tic within a demo). To use WADs it would require that at the end of the demo, a pointer back to the index. If this 0x80 behavior is kept then it will place severe restrictions on the actual format of a demo (if it contains 0x80 at the right position then it fails). Some demos for ports such as ReMooD, Odamex, Zandronum, and ZDaemon use packet based formats of arbitrary sized data.

PrBoom(+) if it switches to a ZIP, can still maintain the WAD inside of it for backwards compatibility with older versions of itself, since ZIP entries can still point to individual entries in a WAD. That is, you can have WAD/ZIP hybrids which point to the same data, you can also have the same ZIP within a ZIP (quine).

[fraggle]

Didn't realise PrBoom reuses the WAD format for metadata. That sounds like a good solution. None of the advantages you describe sound like particularly compelling or necessary. It's just metadata.

[RestlessRodent]

fraggle said:

Didn't realise PrBoom reuses the WAD format for metadata. That sounds like a good solution. None of the advantages you describe sound like particularly compelling or necessary. It's just metadata.

I feel that using WAD would be a step backwards due to its limitations. The 8 character limit puts a strain on the names of lumps. Non-standard lumps should be in their own namespace. Using markers such as BEGIN and END, would require lump iteration rather than a simple get by name. Due to the names of potential ports, there would need to be at least two characters dedicated to a port, followed by a marker to indicate that is is port specific (say GZ_ for GZDoom, PP_ for PrBoom+, PR_ for PrBoom, RM_ for ReMooD, ZA_ for Zandronum, ZD_ for ZDoom, ZE_ for ZDaemon, etc.) which would then give 5 characters to identify what it is actually for. So instead of "x-remood-splitscreen", you would instead have "RM_SPLTS".

[Linguica]

I don't know if you have seen https://www.doomworld.com/vb/doom-editing/72683-idea-metadata-lump-for-slade-and-doom-builder/ but it in all its glory seems relevant.

[dew]

As a person recording demos, sell this to me. Why do I want this? How do I use this? Is it even remotely possible for zdoom and c/s ports to follow this standard when all of them use their own demo specs and quite possibly don't need any hacky extensions?

[RestlessRodent]

dew said:

As a person recording demos, sell this to me. Why do I want this? How do I use this? Is it even remotely possible for zdoom and c/s ports to follow this standard when all of them use their own demo specs and quite possibly don't need any hacky extensions?

A supporting source port would automatically add the metadata to the demo you recorded. It could be used to contain information as when you recorded a demo and its length, so you can for example take your demos and categorize them if you want.

It is very possible for ZDoom and C/S ports to follow this standard. This metadata would be port agnostic and should not interfere with the actual demo information used by a port. The purpose of this metadata is to work with the various demo formats for each port since they do vary widely. One major requirement is that there is an end of demo specifier, otherwise the metadata may be treated as demo data (Doom/Boom have 0x80, ReMooD has DISC, ZDoom has DEM_STOP, Odamex has svc_netdemostop). The major advantage is that you can identify details of your demos without the need for separate utilities for each demo format that the various source ports have. So instead of say launching a source port to see which WADs were used for it, or guessing which WAD a vanilla-compatible demo was recorded with, you could just do a (an example on Debian) `unzip -c yourdemo.lmp ddmd/files` which will list the WADs you recorded with.

I would say a hacky extension would be inventing a whole new format and placing it somewhere, whereas ZIP is not rather hacky and can exist at the end of any file.

This could also simplify uploading/processing at DSDA if it uses such metadata as it can fill in information automatically when uploading.

[dew]

I've parsed your wordy post for stuff important to people who'd actually use this in practice and I got:

- time of recording
- demo length

You must be joking. pr+ and the C/S ports already include metadata to identify wads that were used to record a demo, either in the footer or by changing the lmp header spec entirely, so you're merely trying to tell them to change their standard to yours, which is NOT going to work.

I'm going to be blunt: you link to a thread that wants text messages or some other form of commentary recorded in demos without breaking vanilla compat. Can you do this with your implementation and how?

[fraggle]

If you're creating something entirely new then to be perfectly honest I would have gone with something far simpler, like some JSON data just attached to the end of the file for example. That gives you a structured format that's far more useful.

Either that or something like RDF which is designed for precisely this kind of purpose.

I don't think the embedded zip file solution is a good one - it's simultaneously overcomplicated while not actually solving any real problem as far as I can tell. In general I think it's better to standardise existing practices than invent entirely new ones anyway, so working with the existing PrBoom format - which is already extensible - seems nice.

[RestlessRodent]

dew said:

You must be joking. pr+ and the C/S ports already include metadata to identify wads that were used to record a demo, either in the footer or by changing the lmp header spec entirely, so you're merely trying to tell them to change their standard to yours, which is NOT going to work.

For specific ports:

• There is no WAD information at all.
  
• The WADs used are in the demo header.
  
• The WADs are not in the demo header, but are instead part of a "change of wad/map/level" packet.
  

Also, I am not changing anything about existing demo formats at all. This extra metadata would instead just be placed after the demo. The demo formats which I have looked at (Vanilla, Boom, Odamex, ReMooD, and ZDoom) via source code, do not appear to care at all what appears after the end of demo marker. This standard idea is just exploiting the fact they the ports do not care what is after their end of demo marker. So the only changes to the code that would be required is appending the metadata after the demo data when it is written to the disk.

If you take a demo you recorded in vanilla, then perform 'cat foo.lmp doom2.wad > bar.lmp'. You can still play bar.lmp back as if it were foo.lmp (however if the metadata is too large then vanilla will refuse to play it due to out of memory, which is why ZIP compression would work here).

PrBoom+ also only records (part of) the command line which was used to record the demo with, this specific "wads" that this standard would use would just list the WADs alone with their identifying marks.

dew said:

I'm going to be blunt: you link to a thread that wants text messages or some other form of commentary recorded in demos without breaking vanilla compat. Can you do this with your implementation and how?

Yes. Commentary or chat text would be placed within a contained entry in the trailing ZIP/WAD which may then be extracted by ports able to read and handle the commentary. This works without breaking compatibility with Vanilla because the associated data is after the actual demo data. Vanilla demos are completely linear. Exploiting the same thing which allows you to have EXE/ZIP hybrids.

fraggle said:

If you're creating something entirely new then to be perfectly honest I would have gone with something far simpler, like some JSON data just attached to the end of the file for example. That gives you a structured format that's far more useful.

That would limit metadata to non-binary.

fraggle said:

Either that or something like RDF which is designed for precisely this kind of purpose.

I am assuming you mean https://en.wikipedia.org/wiki/Resource_Description_Framework

fraggle said:

I don't think the embedded zip file solution is a good one - it's simultaneously overcomplicated while not actually solving any real problem as far as I can tell. In general I think it's better to standardise existing practices than invent entirely new ones anyway, so working with the existing PrBoom format - which is already extensible - seems nice.

Either WAD and ZIP can work (ZIP is more powerful however, as previously stated). However the method which PrBoom uses to discover the WAD in the first place would be broken in most every source port.

So of WAD and ZIP, they are placed in the following way:

• For ZIP: The end of central directory is at the end of the file.
  
• For WAD: After the end of the actual demo content: 1) WAD starts on a 4-byte alignment, where the length is padded to 4 bytes; 2) A 32-bit unsigned integer specifying the size of the WAD file (with its 4 byte padding at the end); 3) A 32-bit unsigned integer containing a magic number (say 44 64 4d 44 (big: 0x44644d44, little: 0x444d6444)); 4) A CRC32 checksum of the WAD (with its padding).
  

Appending a WAD without its checksum and length would just be suicidal as there would be no sure fire way to actually determine if it is valid or not. Using a WAD would require a tool to read metadata from the demo or extract the WAD for use by tools such as DEUTEX and such.

Personally, I would prefer ZIP over WAD, but I am willing to use WAD rather than some other fancy format since it is the most common in Doom. I would have posted a poll, however the topics in this specific forum do not permit polls. It would be best to get some more opinion and insights as to whether it should be ZIP or WAD.

[Xaser]

i will never be able to resist butting into conversations just to link this when it's relevant