Idea: METADATA lump for Slade and Doom Builder

In compiling a lot of resource wads and a few mapsets over the years, I've sometimes run into situations where I can't remember which version of a map is in the compiled wad, or I can't remember which resource wad something originally came from.

Because there is no concept of a "Date Modified" field or other tags for lumps, the only solution is to guess and check, which can be frustratingly time-consuming. I had an idea recently to solve this issue, via a METADATA lump that could be automatically managed by Slade and other tools when editing a wad, and could contain additional data for each lump in the wad.

When a lump is copied from another wad, Slade could include metadata for the filename and Date Modified of the file they originated from. This would enable mapset compilers to more easily determine whether or not they have the most up-to-date version of a given map, for instance. Ideally GZDoomBuilder could also be set up to maintain date-modified tags for the map lumps themselves.

Reusable assets like textures and MIDIs could be set up with additional metadata specifying their authors, so that this information would easily follow them when they're put into other wads. This would make it far easier to track down the origins of resources, even from a second- or third-generation copy that didn't have a thorough credits list.

Because Doom ignores unknown lumps (unless you're trying to load them as a map/graphic/etc) this would be a completely vanilla-compatible way to extend the wad format to be more editor-friendly.

Thoughts? I'm curious to hear developer opinions on this, and there are probably a lot of other usage possibilities I haven't thought of, too.

1 person likes this

Share this post


Link to post

A very reasonable idea. I'd imagine that if it were implemented at least once (in an editor), then hopefully it would get more people implementing it in various other editors. It would be nice to eventually see it spread to a fork of DB2 for those who are somewhat opposed to GZDB. How would you propose the metadata is formatted?

Share this post


Link to post

Storing metadata such as timestamps would probably be better in its own structure (like a second directory) than a lump, because the lump can be, well, edited like a lump. The WAD format allows that because it doesn't actually care if there is space not addressed by its directory.

For MIDI I'm personally favorable to using MIDI metadata. In fact I've even added to the devbuilds the functionality of displaying MIDI metadata (according to the types as listed here).

For texture there's no real solution to embed the metadata inside the file. That's possible with some formats like PNG but since it is not universally supported it's not fool-proof.

For maps you can actually put some revision info in the map label, heh.

Ultimately though I think for converns of date tracking and versioning the best would be to use a dedicated versioning software like Git. Trying to replicate these features by bloating a wad file with voodoo metadata would be a rather haphazard solution, not as practical as what already exists, and also a massive undertaking.

Share this post


Link to post

The main idea would be for a metadata lump to be supported by the major WAD editing utilities like SLADE and GZDB, and just be handled automatically. It wouldn't be critical for anything, just provide a relatively standardized way to include information in a WAD file. The most obvious way would be to list stuff by lump name, and then just add whatever tags and fields you wanted.

Like, I dunno, if it used YAML:

MAP01:
    author:		Joe Blow
    title:		My First Level
    last-modified:	1000197960
    description:	blah blah
    
TITLEPIC:
    author:		Joe Blow
    last-modified:	1000197961
    description:	My face as a titlepic!!
etc. Whatever engine you were using wouldn't care what was in the metadata lump - it would just automatically be created / read by SLADE et al as a way to store pertinent info if you so desired.

Share this post


Link to post

Doomsday introduced the Info file format for addon metadata with the frontend (Snowberry) back in 2006. Although there aren't a huge number of addons that use it, there are some big resource packs that do.

Share this post


Link to post

Please add support for TAGDESC and PLATFORM too! :)

Share this post


Link to post
Gez said:

For texture there's no real solution to embed the metadata inside the file.

The patch format which DOOM uses could store some additional data -- place it after the offset values and before the pixel data.

Share this post


Link to post

In my experience, I've asked mappers to use specific naming conventions for my own convenience when sending me different versions of the same wad file and I keep all individual maps in a folder and out of a compiled wad until a few days before release. That works fine for me.

Share this post


Link to post

What we really need is a platform, for which I suggest the apt name of Trusted WAD Integrity Technology. The WAD directory must contain a signature and public key file for the asset author, who has registered with the platform maintainer (for a $100 annual account maintenance and licensing fee, of course), putting the corresponding private key in escrow along with a registry of hash values for WAD files approved to utilize the contained signature and its resources.

On loading such a WAD file, whose other entries are encrypted with the private key, the implementing source port must contact the platform server over a HTTPS connection to retrieve the private key for those resources. If the WAD file hash is not registered for use of said resources, the entire program will be aborted.






</sarcasm>

Share this post


Link to post

I'd rather like to see a standardized format (maybe similar to UDMF) to let tools/users store arbitrary data in such a lump. Lump metadata could be a subset of this format.

Quasar said:

dangers of DRM

At least the ports and tools are open source, so you could always remove that code (or would that be illegal, then?).

Share this post


Link to post
andrewj said:

The patch format which DOOM uses could store some additional data -- place it after the offset values and before the pixel data.



That will make some sanity checkers reject it because they assume that the first column immediately follows the column directory. ZDoom would reject such a patch as broken.

Share this post


Link to post
Memfis said:

Please add support for TAGDESC and PLATFORM too! :)


What exactly did these lumps do? I've seen them in some older WADs but had no idea what their purpose was.

Share this post


Link to post

This may not be exactly what was asked for, but GZDoom Builder can create a text file in the script editor.
This is saved separately from, but in the same folder, as the map. As long as the text file remains in that folder,
it can be recalled and added to.

Share this post


Link to post
Glaice said:

What exactly did these lumps do? I've seen them in some older WADs but had no idea what their purpose was.


They were also metadata lumps stored by specifid editors -I only recall seeing PLATFORM though, and I don't remember which one stored it.

Of course, a NEW and ENHANCED metadata lump for post-Y2K Doom must use NEW and ENTERPRISEY data formats, such as XML, JSON, etc. *grin*

Share this post


Link to post
Glaice said:

What exactly did these lumps do? I've seen them in some older WADs but had no idea what their purpose was.


They were used by Geoff Allan's DoomEd, so you can find these lumps in the accompanying example levels.

Share this post


Link to post

It's a good idea, but no one will support it properly. Here's what I'd like to see:

. Long filenames
. lump date/time
. lump MD5
. lump author/description, texture group (from an approved/agreed-upon set of groups), etc
. map author/description
. map requirements (jump/crouch, large blockmap, intercept passthru, archvile ghosts, 3d floors, flats on walls, and 1,000 others), additional PWAD requirements, additional port requirements, etc.
. REJECT edits (to make sure map editors don't rewrite hacks)
. WAD-level author/description
. so much more

Yes, I went way beyond the OP's idea. The only way to get somewhat universal support for ANYTHING ranging from OP to my idea is for someone to:

1. Write an entire spec.
2. Design it, and create some working example WADs.
3. Write a C library that can be dumped into source ports as-is.

I would only be interested if the METADATA was very encompassing, easy-to-use, extensible, logical, and compatible with all manners of ports, editors, WADs, PK3's, etc.

Point being, it has to provide substantial benefit, in a way that cannot be easily done otherwise. For the OP's idea, yes, editors could maintain a lump/datetime stamp in a text lump rather easily.

Share this post


Link to post

I've been thinking about adding something like this to SLADE eventually, so yeah, I'm all for it if something standard can be agreed upon.

I think the best (and simplest) way to store it would be just as a regular lump - hiding it in the wad file somewhere not referenced by the directory is fine for wad files, but wouldn't really work for zip/pk3.

As for the format of the data itself, I'm open to anything really, as long as it is reasonably flexible like UDMF.

Share this post


Link to post
sirjuddington said:

I think the best (and simplest) way to store it would be just as a regular lump - hiding it in the wad file somewhere not referenced by the directory is fine for wad files, but wouldn't really work for zip/pk3.


Thing is that zip/pk3 don't need a new way to keep track of metadata such as time stamps -- it's already covered by the zip format.

Then it's a question of deciding exactly what metadata is stored, so as to see if there's a reason to implement that for zip/pk3 as well.

Share this post


Link to post
Gez said:

Thing is that zip/pk3 don't need a new way to keep track of metadata such as time stamps -- it's already covered by the zip format.

Then it's a question of deciding exactly what metadata is stored, so as to see if there's a reason to implement that for zip/pk3 as well.

If you ignore what I wrote, it'd be rather simple, which may be a good thing.

Share this post


Link to post

Since WAD lumps can be reasonably assumed to have unique names, I still think that the easiest way is just to have a human-readable format, be it YAML or XML or JSON or whatever, where you can assign arbitrary keys/values to a given lump name. SLADE (or whatever other utility) would have an internally defined set of common ones, like a timestamp, which would be automatically created / updated when lumps were created or imported. Users could then manually set other common ones (author, description, etc) or could make their own arbitrary keys. KISS.

Share this post


Link to post
kb1 said:

It's a good idea, but no one will support it properly.

If it's supported by SLADE and GZDoom Builder (i.e. the DB branch that's still under active development), that's enough to make it really useful, I think! I'm excited that sirjuddington sees some use potential in it.

If it can be made more advanced and encompassing than that, great! But even a simplistic variation supported by only one or two common editing tools would be so incredibly useful that I don't want to see this fall to the wayside due to overcomplication like the early attempts to create what eventually became UDMF.



@Gez: I understand that there are ways of manually putting custom arbitrary info into a wad file, and sophisticated version-control systems for keeping track of files with other team members. But being able to go back to a resource wad you haphazardly compiled during a crazed mapping fugue several years ago and instantly be able to find out where you got a certain asset from, just because the lump editor was smart enough to automatically store that information for you, would be awesome.

Share this post


Link to post
Linguica said:

Since WAD lumps can be reasonably assumed to have unique names

Nope, they cannot.

The best example is any map lump. THINGS, SECTORS, etc. are all identically named.

But beyond that it's also possible to have the same name for different things; for example a patch and a flat can have the same name, for instance. That's where the namespaces are useful.

And even further, it's also possible for names to be duplicated entirely. A good example would be doom.wad, in which each of the sounds and music track from the shareware episode are duplicated (first you have the sound and music from KDITD, and then you have the sounds and music from the entire game, including KDITD, TSOH and Inferno).

There are other examples. 1 Monster features two different D_SHAWN songs notably; and it's quite common to have duplicated marker lumps (I've seen wads which had messed-up organization like F_START lumps lumps lumps F_END S_START lumps lumps lumps S_END one map why not F_START more lumps F_END S_START again more lumps S_END etc.


So yeah, wad lumps cannot be assumed to have unique names. By design it doesn't happen as soon as you have more than one map in a wad; and by mistake it happens quite frequently as well.


I'll even go further and say that even in zips you cannot assume each file will have a unique name. Most zip utilities will overwrite and only allow one file per file name; but the zip format itself doesn't require that and it's perfectly possible to have a zip with a bunch of files all having the same name. You can create one in SLADE 3 or SLumpEd if you want.

The one thing that can be relied upon is order -- two different files cannot be at the same position in the directory. (However, a single file can be at several positions in the directory, with different names...)

Share this post


Link to post

I'm interested to hear peoples thoughts regarding the existing implementation of an extensible add-on metadata system in Doomsday/Snowberry. Is it not suitable for what is being asked for here? Why reinvent the wheel when an established solution exists?

Share this post


Link to post

For real success, whatever method used should have the following properties:

1. It must be able to survive load+save from an unsupporting editor. That suggests that it must be a lump.

2. The lump should not really be able to be created by the user. That suggests a lump name containing symbol characters, or at least above normal ASCII range (>0x7F), as a way to avoid someone making a lump with the same name. Alternately, the lump could contain a "magic" header.

3. It would be nice if it stored size, and MD5, or even CRC-32, so a supporting editor can verify that the lump being described now is the same lump that was originally described. (Maps would require special treatment) Otherwise, it's data cannot be trusted. Please note that, if wads start having this data, programmers are going to make their ports use this data (show author name during game load??). Extra care must be taken to ensure the lump is believable, and that must happen in version 1.0.

4. The software should be able to tolerate missing lumps. It would be nice if a non-supporting editor could merge two wads with METADATA lumps, yielding 2 METADATA lumps, and the metadata keeps working for all entries. That may be too much to ask. It is not foolproof, but, by and large, it would work more often than not, so it should be supported, I believe. Supporting editors would, of course, combine multiple METADATAs into one, preserving lumps with a matching name/MD5/size combo, and purging non matches.

Think of how Windows handled and supported 8.3 filenames in a backwards-compatible manner. That worked 99.999% of the time. Checking MD5 stamp would be even better.

Note - there may be more reliable ways to do it. However it's done, those above points would constitute a minimum requirement for me, cause otherwise it'll start to suck when people use the wrong tools.

Share this post


Link to post
DaniJ said:

I'm interested to hear peoples thoughts regarding the existing implementation of an extensible add-on metadata system in Doomsday/Snowberry. Is it not suitable for what is being asked for here? Why reinvent the wheel when an established solution exists?

Honestly I think the onus is on you to explain why a language/syntax/whatever specifically designed for addons to a single source port (which has long since abandoned any notions of interoperability) would be the natural choice for a feature meant to be backwards compatible and source port agnostic.

Share this post


Link to post

Well, I did provide links to all the documentation for the format, which presents examples of its use both in documentation and as downloadables. I'm not sure there is really much more I can add here in a forum post that isn't already covered elsewhere.

While this format was certainly designed with Doomsday's needs in mind, there are no direct ties to this particular engine.

I am more than happy to discuss the format with everyone here. At the end of the day it benefits the whole community if a single standard is achieved and from my perspective it would seem to make a lot of sense to adopt an established mechanism which already has a working implementation.

Re: long since abandoned any notions of interoperability

That is not true. In fact in recent years supporting the existing features of other ports, like MAPINFO (coming in 1.15) and map hacks, has been a driving factor in development (and will continue to be a focus in coming versions).

I do agree that for a long time Doomsday has gone its own way with regard to user-made content however this is largely because we tend to focus on different aspects of modding (3D models, lighting etc...).

Edit: Dangnamit it seems the docs weren't updated following the last wiki migration. Bear with me, I'll try and whip them into shape...

Share this post


Link to post
kb1 said:

2. The lump should not really be able to be created by the user. That suggests a lump name containing symbol characters, or at least above normal ASCII range (>0x7F), as a way to avoid someone making a lump with the same name. Alternately, the lump could contain a "magic" header.

3. It would be nice if it stored size, and MD5, or even CRC-32, so a supporting editor can verify that the lump being described now is the same lump that was originally described. (Maps would require special treatment) Otherwise, it's data cannot be trusted. Please note that, if wads start having this data, programmers are going to make their ports use this data (show author name during game load??). Extra care must be taken to ensure the lump is believable, and that must happen in version 1.0.

I pretty strongly disagree with these sentiments. I don't think that trying to discourage making a metadata lump hand-editable is a good idea, for a couple reasons:

* It would be impossible to stop people from "forging" information if they really wanted to. "Security theater" is counterproductive.

* Trying to make it tamper-resistant would just increase the urge for people to treat it as some sort of canonical record, which it could never be, instead of just being a mildly useful feature.

* It would restrict creating / editing metadata lumps to the handful of tools that supported it, which would preclude people using other tools, or on other operating systems, from getting any benefit from it.

* Re: source ports, I think that's unlikely, since source ports interested in adding that sort of basic stuff already have solutions in place, see, e.g., DED, DDF, MAPINFO / ZMAPINFO / EMAPINFO, etc. And if a port wanted to, then, uh, cool? Why not?

* Perhaps most importantly, a deliberate goal of trying to discourage unrestricted editing / hacking of WADs is entirely antithetical to the ethos that spawned the Doom WAD scene in the first place.

Share this post


Link to post

I think I mostly agree with Ling, though a field for tools to store an MD5 hash (or whatever) would have a useful function regardless, as a counterpoint to the date-modified field. If the MD5 doesn't match, then the program reading knows that the listed date-modified is the last time it was modified in a METADATA-friendly tool, but that it was modified elsewhere at some point afterwards. Since inevitably older Slade/DB versions (and of course other tools) will still be around and in use for some time, this sort of check would help make the data more useful for version-checking on group projects.

I definitely agree that it would be needless to use lump metadata for anything mapinfo-esque, as source ports already have formats and mechanisms for defining data that's relevant ingame. The intent here is to add missing functionality to wad editing tools, and it probably shouldn't get feature-creeped into something more complicated than that.

Share this post


Link to post
DaniJ said:

Dangnamit it seems the docs weren't updated following the last wiki migration. Bear with me, I'll try and whip them into shape...

This is taking longer than I anticipated. I distinctly recall that the last time I looked at these docs, they were of a much higher quality than they were before I started the clean up. (Not sure quite what happened there).

Clearly these docs are presently insufficient for a proposal to adopt this format (or much use to addon authors!). I will continue to work on improving the docs, scraping the parser for specifics and picking skyjake's brain to resolve any ambiguities. Might take a day or two, though. My apologies.

Edit: I too am in complete agreement with Ling for precisely the same reasons.

Share this post


Link to post
Linguica said:

I pretty strongly disagree with these sentiments. I don't think that trying to discourage making a metadata lump hand-editable is a good idea, for a couple reasons:

* It would be impossible to stop people from "forging" information if they really wanted to. "Security theater" is counterproductive.

It's not about discouraging anyone - it's about being able to have some faith that the data is reliable. Imagine if it were a simple text lump, edited by hand:

CRATE1,kb1,2015-05-15 14:22:03,12345
CRATE2,kb1,2015-05-15 14:28:12,14103

Then, someone else adds a new texture, and manually edits the METADATA lump:
CRATE3;Ling;5/15/2015 3:08pm,12k

This data is now useless with its mismatched formats. Now, let's say someone else modifies CRATE2, and forgets to edit the METADATA lump.

The original CRATE2 METADATA entry is now in the expected format, but totally wrong. This possibility cannot be prevented if people are allowed to edit the text lump. But, that's not the only way METADATA can get corrupted. Imagine merging 2 WADS into one. Now, you have 2 METADATA lumps to deal with. And, there are other scenarios too.

I am not suggesting that the METADATA lump (or whatever it gets called) be made uneditable by giving it a funny name, or by some other mechanism. Yes, people should be discouraged from editing the lump manually. But, the "funny name" was to ensure that the name is completely unique across all WADs. More on that below.

Now, my whole idea was to build this data block in such a way that ensured that it could survive being opened in an editor that knows nothing about METADATA, and be recoverable when a knowledgable editor is used. That requires a hash mechanism, and that suggests that the computer should be the one editing the METADATA lump, not the user. Who wants to type a 32-character hex number?

The idea is: When the editor opens the WAD, it reads any and all META lumps, and performs the lookups against all lumps, virtually attaching the meta properties to the lumps, in memory, if the name, MD5 stamp, and size exactly match. This meta data is adjusted in memory as the WAD is being edited. Upon save, the program generates a new METADATA lump, and includes it in the WAD directory.

Because of the lookup/MD5 match step, any hacks done by non-conforming editors would be detected, and the metadata for those lumps would be invalidated, leaving the others intact. Thus, the metadata partially or fully survives such an action.

So, yeah, users should not edit this lump manually.

Linguica said:

* Trying to make it tamper-resistant would just increase the urge for people to treat it as some sort of canonical record, which it could never be, instead of just being a mildly useful feature.

* It would restrict creating / editing metadata lumps to the handful of tools that supported it, which would preclude people using other tools, or on other operating systems, from getting any benefit from it.

We can't prevent *urges*.

"Tamper-resistant" suggests deliberate actions. I would use the term "error-resistant" to describe my idea. In my mind, the *only* way this metadata is useful is if it can be trusted to be accurate, after having opened the WAD in 10 editors, after merging WADs, splitting WADS, etc.

It's not a CREDITS lump. The idea is to be able to capture specific additional lump properties, in a deliberately restrictive manner. That's the only way an algorithm can use the data.

Now, I don't know what would entice anyone to "hack" this data, or what benefit that could possibly provide. Regardless, the point was not to make the lump uneditable specifically, but rather to prevent name collision. "METADATA" might be used by some other tool, but "MŒ±ÅÐűÅ" would not :)

Now that I look at that, that is pretty damn ugly! Maybe "~~META~~" would be sufficient. You don't want the editor mistaking it for a texture, and you don't want to use a name that might be used for other reasons by some ports. For the longest time, my home port would read a lump called "GAMEINFO", which was nice until ZDoom created their own "GAMEINFO" functionality :( (I really wish devs would collaborate a bit on special names).

Linguica said:

* Re: source ports, I think that's unlikely, since source ports interested in adding that sort of basic stuff already have solutions in place, see, e.g., DED, DDF, MAPINFO / ZMAPINFO / EMAPINFO, etc. And if a port wanted to, then, uh, cool? Why not?

Maybe you're right.

Linguica said:

* Perhaps most importantly, a deliberate goal of trying to discourage unrestricted editing / hacking of WADs is entirely antithetical to the ethos that spawned the Doom WAD scene in the first place.

Come on, man... I have to call "bullshit" on that one.

When you see an OGG file in a WAD, are you upset that the editor doesn't provide a hex editor to let you hack away at the bytes? It's kinda the same thing, if you think about it.

I would expect compliant ports to generate something like this:
Name/User/DateTime/Size/MD5/Editor
CRATE1/kb1/2015-05-15 14:22:03/12345/1BC2C2E00B4058F6067EA3E9ABA02A65/WadHacker 2.0


You could even go further, and support additional custom name/value pairs:

For textures: Original WAD name, original lump name, texture pack name
For custom monsters: Source web site, monster name
For maps: Author, port requirements, feature requirements

And, yes, if adopted, ports could maybe get some use of it.

My final point: If it is done properly, METADATA can be used to emulate a backwards-compatible, built-in extension of the WAD format that can survive being edited by an old editor. That's impressive and powerful.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now