Idea: METADATA lump for Slade and Doom Builder

[DaniJ]

Altazimuth, I do agree that on that basis this does appear to make sense. However, if instead of:

(Zip file)
~META~
{
  textures/texture1 { author: Altazimuth }
  textures/texture2 { author: Altazimuth }
  textures/texture3 { author: Altazimuth }
  textures/texture4 { author: Altazimuth }
}
textures/
  texture1
  texture2
  texture3
  texture4
  ...

You use something like:

(Zip file)
textures/
 ~META~
 {
   author : Altazimuth
 }
 texture1
 texture2
 texture3
 texture4
 ...

You then remove the need for a specialist tool to automate the donkey work entirely. It becomes far easier to simply open a text editor and add whatever metadata you want, where you want it.

[esselfortium]

I think you're still missing or ignoring the point of what the automation, and with it the entire feature request, is actually for.

[DaniJ]

So what is the intended point of automation here? If the idea is to simply avoid editing the "raw" files and instead hide this behind a fancy GUI - you can still do that without preventing people who don't want to work that way and mandating the use of your fancy GUI. I do not agree that we need to significantly inconvenience one camp to satisfy the other.

[esselfortium]

The entire purpose of the feature request is to have additional data that is automatically *moved* alongside a lump when it is copied from one wad to the next, so that the data follows the lump from one file or project to another. If you're not using a tool that does this for you when you're assembling your lumps, then you're not getting any more benefit out of it than you would by just writing a textfile like you would do currently.

[DaniJ]

How is that not achieved by simply moving a folder and its contents to another place? (In the aforementioned, alternative model)

[esselfortium]

I guess you could, but metadata isn't intended to be tied to a group of lumps. It's tied to a single lump. You don't need to copy the container that the lump is in, it will always follow any of the lumps.

Plus, if metadata is defined for entire groups rather than individual lumps, then how can lump timestamps be implemented in WAD format, without adding redundancy by defining both the individual lumps for timestamps and then the groups for everything else?

[DaniJ]

I have already demonstrated how timestamps can be included in WAD. By using a solution like I proposed - this information is included in the WAD but excluded from the ZIP, thereby removing another reason for requiring a specialist tool to automate this work.

Please understand that if one addresses all of these issues in the design of the solution, you can then drop the requirement for hashes and timestamps in ZIP and thereby dispense with the need for specialist tools entirely.

It doesn't matter whether metadata is intended to be tied to individual "lumps" if you use the model I suggested - you get the option of a 1:many relationship even in WAD.

Edit: To clarify - I accept the need for a specialist tool in the case of WAD, this is unavoidable. However, with some careful planning we can avoid it in the case of ZIP.

[Gez]

DaniJ said:

Please understand that if one addresses all of these issues in the design of the solution, you can then drop the requirement for hashes and timestamps in ZIP and thereby dispense with the need for specialist tools entirely.

Timestamps in zips are redundant with the already-existing basic features of the format. But the need for hashes remain.

A blanket application of metadata to an entire directory suffers from some logical problems.

First, it mandates a certain type of organization, with files sorted in folders depending on a metadatum, and this is not necessarily what the user wanted. With your idea, I have to get (for example) textures/esselfortium, textures/nickbaker, textures/olabjorling, etc. instead of textures/techbase, textures/caverns, textures/castle, etc. or textures/map01, textures/map02, textures/map03, etc.

The second, which logically follows from the first, is that it turns the metadata into a metadatum. Because unless I try something like using only techbase textures by esselfortium for map01 and only cavern textures by Nick Baker for map02, etc. I can't do blanket application of several different criteria on all the files within a subdirectory. Or I'll have to do that by nesting subdirectories into subdirectories, which is not necessarily desirable. Among other things it makes browsing less convenient, because for example I'd have to navigate up and down folders to compare Nick Baker's techbase textures with Ola Björling's.

Now suppose one of my collaborators working on the same project just imported a bunch of textures into the first folder available, without really caring about who made them. Because metadata is applied to everyone, these files get a wrong attribution. With a 1:1 attribution, they would have no attribution, which I feel is preferable to an erroneous one.


Finally, you complain that the approach taken is wad-centric, but what you suggest is basically a zip-centric approach. And the problem here is that the problem is largely irrelevant to zip files. When you use a zip file, you can use:

• Not MUS, but MIDI with metadata, or tracker modules with metadata, OGG with Vorbis comment metadata, MP3 with ID3 metadata...
  
• Not DMX sound effects, but WAV with LIST INFO and bext metadata, or FLAC with Vorbis comments
  
• Not Doom picture format and raw flats, but JPG with EXIF metadata or PNG with the same thing in a zTXt chunk or even a custom metadata chunk

And so on. When using industry standard formats (rather than proprietary game formats designed at an era where fitting all the content on a single floppy disk was desirable), all the metadata we want is generally already there. The real problem is when we can't use them. Which means targeting vanilla or Boom compatibility, rather than Doomsday/ZDoom, so that zips become irrelevant.

This has to be a wad-centric design because there is basically no real need for it in a zip archive.

[esselfortium]

I think my biggest question here is: Why would someone who doesn't want to use Slade, want to spend the time to manually create metadata that has no relevance to source ports and literally only exists to be displayed alongside the lump list in Slade or similar specialist tools?

If you don't want to use a tool that displays metadata, you're not going to get any perceivable benefit from the time investment of creating it, no matter how much the format bends backwards to accommodate this. The whole argument is based on a flawed premise.

[DaniJ]

I'm not following on the requirement for hashes in ZIP. Where is this necessary as means for relating metadata to file(s).

There are certainly some logical problems considering a 1:many application at a folder level as the only option. However, that is not the argument I am making. In my proposed solution a 1:many relationship is an option like 1:1 is an option.

I fully realize that this adds some complexity to the model but in exchange the mod author gains the freedom to structure their mod (or, subsets of it) according to whichever relationship fits their requirements best in that specific case.

Yes there is indeed a chance that metadata can be incorrectly assigned in the specific situation you describe, however the argument I would make here is that is in an integration task. It is the responsibility of the person performing the integration to ensure that metadata remains correctly attributed in this situation and therefore should reflect on the structure chosen for their mod. For example, if I know I am going to be integrating textures from multiple authors then I will structure my mod to make this task easier, allowing me to simply move the metadata file supplied by the contributor and their textures into the relevant folder in my packaging of it.

There is absolutely a case for using specialist tools to make this process easier but I argue that in a great many cases, a model like I propose can be sufficiently managed simply using a folder hierarchy and a text editor. In particular with the simple cases of a mod made by one author, or, whose logical components are provided by different authors (e.g., maps by essel, textures by Nick, etc...).

The main point I am making here is that these are all design decisions that change according to circumstances and therefore the composition and structure of the mod should reflect that. This is also true in a metadata sense. Consider the benefits of hierarchically attributed metadata. If we limit mod authors to a 1:1 relationship between metadata and individual files we force these structural decisions to be considered at every stage and each time a new resource is added. Certainly, a tool could be used to address these with different "policies" to constrain what is possible but then we loose the other benefits.

And this is the crux of my argument. We dispense with many different possibilities by blankly applying a database and a 1:1 model at format level. To retain any of this logical structure the mod author has chosen - we would have to include this information in the metadata as well (yet more redundancy).

I accept that you see this as the logical opposite of a WAD-centric approach, however, I don't agree that is actually the case. My argument applies equally to ZIP, to the native file system and basically any other file container you might think of.

And principally, the metadata should not be overly concerned with the type of datafile it is describing. Not in a general case, at least. Your point that this has to be WAD-centric is based on the assumption of what one might wish to place in the container.

[DaniJ]

I *think* that at this point I have (finally) articulated my case and the other factors behind my argument in a concise manner. If I have not managed to convince as to the benefits of this model I do accept that we simply do not agree on the matter and likely never will.

Please forgive my conduct during this discussion and the somewhat underhand tactics I have employed at times. However, I felt it necessary to convince people to actually consider the points I was trying (and to a large extent, failing) to make.

Hopefully people now appreciate why I could not simply re-produce quality documentation for what is seemingly a simple problem.

[Gez]

DaniJ said:

I'm not following on the requirement for hashes in ZIP.

"Do we need to validate that the metadata corresponds to the same file?"
Yes -> Then we need a hash even for zip files
No -> Then we don't need a hash even for wad files

There are no situations where "we need a hash for wad files but not for zip files" makes sense.

[DaniJ]

When you say "same" file, what is your meaning? This implies an action taken on it, i.e., it has a previous state. In my understanding this would fall under the category of integration, which is a task performed at that time only and can be therefore solved dynamically, either in a specialist editor or resolved by the author manually when working outside of one.

Re: "we need a hash for wad files but not for zip files"
That was not my meaning. If a hash is required for one container then I agree it applies to both.

[jmickle66666666]

a hash is required for WAD files because there can be two lumps with the same name, and using a hash is the only way to distinguish them in metadata.

[Linguica]

jmickle66666666 said:

a hash is required for WAD files because there can be two lumps with the same name, and using a hash is the only way to distinguish them in metadata.

That's not true, you could just list the metadata in the same order as the lumps are listed.

[DaniJ]

Gez, It sounds to me like you are trying to solve a fundamentally different problem. In your case hashes are used to ensure the integrity of the package, working on the assumption it is broken until proven valid.

[kb1]

DaniJ said:

(In reply to a question you directed to me personally)

Well yes. However, it rather depends on what information you plan to place in the "~META~" file and the representation used, as to whether it is viable (or not) as a practical solution to the problem.

For example, will the user be able to simply open a .zip file using their preferred ZIP tool, drop in a new texture they made in their preferred paint package and then edit the "~META~" definitions accordingly, without having to use a specialist tool to rationalize the edit and update this data file on their behalf? If one can't do that - this seems like a significant inconvenience.

You don't edit the META lump yourself - that's the editor program's job. I've stated that at least 3 times.

[Danfun64]

DaniJ said:

essel, there is no good reason why the two must be considered as logically different things. As I have argued from the beginning - this is a short-sighted and WAD-centric view of the world. As soon as you bring a container like ZIP into the picture the whole model being proposed here collapses in a practical sense.

What about those who want METADATA for their vanilla or boom compatible wads?

[DaniJ]

kb1 said:

You don't edit the META lump yourself - that's the editor program's job. I've stated that at least 3 times.

I know. We disagree on whether this is acceptable. My question was intended to highlight the inconvience incurred, when trying to perform a typical editing task with .zip, should your solution (in it's present form) be chosen.

If the intention is to "secure the package by including in it a data signature which can be validated" - I am quite sure we can solve this without forcing every user to jump through hoops in a practical sense by mandating the use of a single, specialist, editing tool.

However, this is not something strictly related to the idea of defining a common standard for metadata. It is a separate concern. We can discuss this here in the process I guess.

Danfun64 said:

What about those who want METADATA for their vanilla or boom compatible wads?

They would not be affected. Metadata could still be included. I am not arguing against the idea, my objections concern how this is achieved.

[Gez]

jmickle66666666 said:

a hash is required for WAD files because there can be two lumps with the same name

Fun fact time! You can have several files with the exact same name in a zip archive.

DaniJ said:

Gez, It sounds to me like you are trying to solve a fundamentally different problem.

Yes, I'm trying to solve the problem outlined in the OP.

You're trying to standardize a way to use folder hierarchy in zip archives.

Fundamentally different things!

DaniJ said:

In your case hashes are used to ensure the integrity of the package, working on the assumption it is broken until proven valid.

It's not the integrity of the package, it's the integrity of any given file in the package. If there is a file mismatch for one entry, it doesn't invalidate the metadata of all the other entries.

And checking whether the entry has been modified is necessary for checking that it's indeed the same entry.

[DaniJ]

Gez said:

Yes, I'm trying to solve the problem outlined in the OP.

You're trying to standardize a way to use folder hierarchy in zip archives.

Fundamentally different things!

No. I presented a solution for associating and delivering a common metadata standard to/with files in a way that maximizes flexibility and which also supports WAD.

What you are presenting is something which requires hashes embedded in the data, to achieve the same result.

[DaniJ]

In the interest of working together to resolve the situation, can we logically separate the problem presented in the OP from both the idea of a common standard for metadata and the solutions presented thus far?

As we have identified, the original problem is a matter of integration. Does this necessarily require embedding hashes in "final products" that will be widely distributed?

[Gez]

Basically your issue is that you are absolutely opposed to the idea of validating the metadata.

[DaniJ]

No that is not the case. I do not principally reject the idea on a fundamental basis. I simply object to the idea of forcing the use of specialist tools to manage metadata which doesn't necessarily require hashing.

[jmickle66666666]

Danij: Why do you keep calling Slade a "specialist tool", that you are suggesting people will be forced to use? Slade is by far the most used, most powerful and actively developed tool for editing wads around. And also, why do you assume that this is the only tool that will support metadata? The entire spec will be open (just like UDMF etc), so any tool can be made to support it. If Slade implements it, then it would make sense for GZDB to implement it too. Any other tool that doesn't support the METADATA lump will not be affected, and there is always the option of editing the information by hand (but as has been pointed out many times, why would you want to?)

[Gez]

He means stuff like 7zip, WinZIP, or WinRAR.

The thing is that his approach is basically a by-humans, for-humans system where you put a readme file in each folder that says "files in this folder where made by Justin Example". Then humans can read the readme, and decide to put more files from Justin Example in the same directory, but files from Ann Otherperson will have to go in a different directory.

The approach we're going with, on the other hand, is by-software, for-software. "User tagged this file as being by Rand Omname, and this other file as being by Sam Wanelse" and it doesn't matter in what folder they are.

[Graf Zahl]

This entire discussion of DaniJ vs. everybody else mirrors nearly all other discussions about design philosophies, i.e. there seems to be a profound phobia for Doom-centric handling among the Doomsday devs.

I remember similar issues about map formats, map format storage and a few other things that all ended with DaniJ trying to discuss the topic to death without even trying to get a consensus with the other source ports and tool developers who are nearly uniformly opposed to the kind of abstraction that seems so prevalent in everything coming out of the Doomsday corner.

[DaniJ]

Ultimately it was only a suggestion. I didn't previously realise the general view, as its never been said. Ok

[Randy87]

So, obviously this metadata will be a singular lump.
This will work with wads/zips/whatever.
It will be textual with fields for what? Just other lumps?
Maybe some kind of compat options that a port could read? Maybe use if the player enables the functionality. (i.e. jumping already enabled/disabled)
Maybe display author names in-game more conveniently?

Compress the metadata lump?
It would probably only need decompressing/parsing at load anyway.

The lump fields would be keyed to a name/hash combo?
Identical lumps (name and data) would just refer to the same data?
Is there any reason someone would need different metadata for identical lumps?
Obviously an indexed design is right out. It lacks robustness and is easily destroyed.

Metadata for maps would encompass all their lumps?
Or simply disallow metadata for zero length lumps, like markers?
How to key textures?

I assume the suggested info about other lumps is just author/date fields for now?
Presumably, new fields or data types could be added down the line without affecting the initial fields?
Old progs/ports would just ignore them. Author, compat, or otherwise.
Generic user-gen'd notes field for lumps?

I don't want to slog through this mess of a thread again, so I don't remember all the listed uses.

[esselfortium]

The primary intent of lump metadata is for display in lump editing tools. Source ports already have their own methods for displaying author info for maps, and compatibility options vary based on different ports' needs, so that's also something probably best kept to port-specific mapinfo lumps or similar. This is more for keeping track of timestamps for in-development projects, and for keeping track of the origins of shared resources as they travel from one project to the next.

There's probably not much reason for source ports to implement metadata reading -- in the immediate future, I'm planning on making use of it for vanilla wads and texture resource packs.