Wad compatibility option database - ideas

[RestlessRodent]

In ReMooD I just have compatibility tiers:

* Vanilla Doom: Pure vanilla style
* Doom Legacy: Doom Legacy defaults (with ReMooD settings disabled)
* ReMooD: The defaults settings I set

[kb1]

wesleyjohnson said:

(lots of important things)

Actually, these are all really good points. The "MAP31 yellow key" thing was something I threw out there - maybe that *is* taking it too far. I do disagree a bit about the Boom compatibility - that *would* be helpful for buggy stairs, for example. But, there's many thing I think we all *can* agree on:

1. This is an important topic to many.
2. It would be very difficult to accomplish 100% success.
3. There are many, many maps out there, and more each day.
4. A large portion of the maps are vanilla, or limit-removing Boom/MBF.

5. A modern source port should be able to be coersed (by properly setting options) into providing a 100% accurate experience on a vanilla, or Boom/MBF limit-removing map (if so equipped). If not, the source port can be considered broken. (Not talking about Chocolate Doom, although, by it's definition, it strives to provide a 100% experience).

With those things in mind, I think we shoot ourselves in the foot to ask for more, at least at first. Sure, we can use a format that can be expanded. Port authors can occasionally submit data to the database in response to bug reports. They can also request a new, port-specific setting as needed. Of course, players can submit data too. It's not that "supporting everything" is too complicated, but, rather that we need to start somewhere.

Once we have some data, port authors will tend to want to agree on what can be considered "standard settings", so that they can take advantage of the data.

Here's some summary points I'd like to make:

1. The vast majority of wads would be served well by options that are common and standard to all modern source ports.

2. We simply aren't going to be able to totally repair all past and future wads - Even if the technical aspects of a format are set in stone, there is just too much data to enter, too much to consider, to expect an overly-complex dataset to become/remain accurate.

3. We should concentrate on the "low-hanging fruit" - the options that are easy to implement, and standard across all ports (Boom/MBF, etc.), and do the best job we possibly can. That will allow us to quickly begin to provide a valuable resource, and have a dataset that, while not always 100% helpful, possibly 75% helpful, on, say 50% of the wads out there.

4. Once we have that partial dataset, we can then focus on improving the accuracy, by focusing on the more-esoteric settings that may be specific to certain ports.

5. There's nothing wrong with having extra fields that aren't (yet) used directly by the port (like DEVPORT). If you believe that that setting is valuable, we can absolutely add it in the spec, and it may be helpful later.

6. As a port author myself (my as-of-yet-unreleased port), I have difficulty considering the needs of other ports (or even keeping up with them). But, I'm trying to step outside of that mentality a bit, and try to define the absolute, most basic, minimal set of standards that we can all gain benefit from.

7. It all started with Linux Doom. That is the most basic standard. And, most ports have adpoted Boom/MBF. That's the next standard. From that point, each port goes in different directions. We would be so much better off by fully covering those 2 standards, and we can get there in agreement. Let's start there, and see what we can accomplish. I want mass support for the first iteration of this thing - we'll need it for it to be successful.

[wesleyjohnson]

I would suspect that of the modern source ports, only prboom and chocolate doom can provide 100% emulation. The rest are not designed to do that, having enhancements and fixes that interfere.
DoomLegacy has a fix for the TNT MAP30 stairs too, but it is not accessed by compatibility flags.

I think the only thing to do at this point is to look at some of these PWAD and record the info, see what it is.

The whole point of DEVPORT, is that would indicate the necessary compatibility flags in one statement. Most PWAD would not require anything more.
The only requirement is that there be some mapping from a DEVPORT setting to what compatibility flags it sets. This is easy to automate. The most difficulty would be those who are interpreting the file themselves and making settings manually.
Then if any specific flag settings are needed they can be in addition,
overriding those flags set by DEVPORT. Most implementations would be order dependent, so DEVPORT would come first, then specific overrides would appear after it.
One advantage of this is that the special flag settings are not buried in all the common boilerplate settings. The appearance of a specific flag setting after DEVPORT would indicate that it is special.

I keep repeating this because it is hard to understand why it would be preferable to copy identical boom compatibility flags for large numbers of PWAD instead. If you are determined to copy the boilerplate boom compatibility settings anyways, then the biggest use of DEVPORT is undermined, as all those settings override the DEVPORT line.

[kb1]

wesleyjohnson said:

I would suspect that of the modern source ports, only prboom and chocolate doom can provide 100% emulation. The rest are not designed to do that, having enhancements and fixes that interfere.

The whole point of 'compatibility settings' is compatibility! If your port does not respect this aspect, it (1) does a bit of a disservice to the whole idea of compatibility, (2) has no business trying to get it's settings from a global compatibility database. Is there really a need for: Doom, Boom, MBF, *and* Legacy stairs, for instance. I'm not trying to dictate what anyone's port should do, but, at some point things get a little ridiculous. I would not recommend it, but, in this case, you could always have a port-specific "legacy_stairs" option. I respect ports having "proprietary" settings, modes, editing features, etc. But, if a port ever expects to be loading a wad that was not made specifically for said port, shouldn't the port handle the experience correctly? Isn't that the whole goal? What I have been proposing all along is a system that provides all the different ports a small bit of extra information that will help it adjust itself to come closer to rendering a positive experience. The issue arises specifically because ports handle things differently - sometimes when they shouldn't. For this whole thing to work, port authors must agree to have the ability to switch to at least some of the agreed-upon modes of operation.

wesleyjohnson said:

DoomLegacy has a fix for the TNT MAP30 stairs too, but it is not accessed by compatibility flags.

But, it could be, couldn't it? Some integration needs to happen anyway, if you plan to support the compatibility db.

wesleyjohnson said:

I think the only thing to do at this point is to look at some of these PWAD and record the info, see what it is.

This is a great idea. I think things will start to make sense when we have some data. We can evolve the specs accordingly.

wesleyjohnson said:

The whole point of DEVPORT, is that would indicate the necessary compatibility flags in one statement. Most PWAD would not require anything more.

I do understand the desire, but it, once again, suggests that lots of settings (many that are not mandatory) will be getting set by choosing a DEVPORT. Of course, the *db website* could have a button to pre-populate checkboxes in response to clicking on a "PrBoom" button, but I would expect to be able to then modify it's choices before committing the record.

wesleyjohnson said:

I keep repeating this because it is hard to understand why it would be preferable to copy identical boom compatibility flags for large numbers of PWAD instead. If you are determined to copy the boilerplate boom compatibility settings anyways, then the biggest use of DEVPORT is undermined, as all those settings override the DEVPORT line.

Yep, that's why I think it's not really needed. I believe that only a couple of settings will be required anyway per map. I have never been suggesting that this system should take away all of the things that make each port unique. Rather, it should ONLY affect those settings that are needed on a map-per-map, setting-per-setting basis. For that to work globally, the settings we decide upon should affect each port nearly identically. The port is not under any obligation to honor the setting, but if it's easy to do so, why not? That's more maps your port will support. If a port honors the DEVPORT setting, it must forcibly set all of those related settings, even ones that are not needed. For example, I typically like the 'corpses sliding off of ledges' setting, but, if I load a "DEVPORT Vanilla" map, that setting will be turned off, for probably no real valid reason.

That's my take, anyway.

[Graf Zahl]

kb1 said:

Is there really a need for: Doom, Boom, MBF, *and* Legacy stairs, for instance. I'm not trying to dictate what anyone's port should do, but, at some point things get a little ridiculous. I would not recommend it, but, in this case, you could always have a port-specific "legacy_stairs" option.

Let's make it simple: If some port's specific quirk is not required by any map in existence it should be removed. Period.

I did so for a few things in ZDoom when it became obvious that they did more harm than good. cessary

[Quasar]

There is no reason that this thread should have gone on for 5 pages. The target audience was lost after the first 2 pages. I haven't been keeping up with this at all and I'd say the probability of me implementing support for anything coming out of it at this point is probably worse than a snowflake's chance in hell.

The onus to demonstrate how this system will benefit any port is on the people suggesting it, particularly if the said same people insist on defining all the details of the implementation on their own despite not being software engineers.

Right now I'm seeing circular arguments, pointless distractions, and what amounts to a grotesque inflation of what should be an utterly simple thing into something of Wagnerian proportions.

[Yagisan]

DaniJ said:Fact is we are nearly three weeks into this discussion and we are no further forward than we were on day zero. [/B]

That is not exactly true. kb1 somehow got both DaniJ and Graf Zhal to agree on something - and Doomsday and GZdoom are almost polar opposites on everything. Given I can't remember when I saw you two agree last, we have determined the solution is what you two have agreed to. End of thread.

[kb1]

Graf Zahl said:

Let's make it simple: If some port's specific quirk is not required by any map in existence it should be removed. Period.

Agreed. Sounds reasonable.

Quasar said:

There is no reason that this thread should have gone on for 5 pages...

Agreed.

Quasar said:

...snowflake's chance in hell...

Aw, come on, man, don't be so negative, besides it's a snowball, not a snowflake. (better chances, more ice) :)

Quasar said:

The onus to demonstrate how this system will benefit any port is on the people suggesting it, particularly if the said same people insist on defining all the details of the implementation on their own despite not being software engineers.

Don't know about everyone else, but I've made my living writing software for 2 decades now. Good software? Well, they keep me around, even when I spend my time on the web, trying to get a few Doom source port devs to agree... But, you're right, some demonstration is in order. Of course, it's a chicken-and-egg problem - hard to demonstrate without a database, hard to get a database without a demonstration...

Quasar said:

Right now I'm seeing circular arguments, pointless distractions, and what amounts to a grotesque inflation of what should be an utterly simple thing into something of Wagnerian proportions.

Strongly agree. It *is* a simple thing, or it could/should be. But a lot of people envision it becoming more - being the "definitive answer" to source port options, and it simply can't - nor should it.

Yagisan said:

That is not exactly true. kb1 somehow got both DaniJ and Graf Zhal to agree on something - and Doomsday and GZdoom are almost polar opposites on everything. Given I can't remember when I saw you two agree last, we have determined the solution is what you two have agreed to. End of thread.

Honestly, I'm surprised it lasted this long. I have already committed to creating a working model for this thing, but, as stated previously, it's going to take me considerable time. Stay tuned.

[DaniJ]

I doubt a working prototype is quite what Quasar had in mind when he asked for a demonstration of benefit. Presumably the fact that numerous source port developers have acknowledged the problem and shown interest in a solution, all without any operational model in place says to me that what we are actually looking for is a demonstration of provability/suitability to ratify the design of a proposed solution.

[kb1]

DaniJ said:

I doubt a working prototype is quite what Quasar had in mind when he asked for a demonstration of benefit. Presumably the fact that numerous source port developers have acknowledged the problem and shown interest in a solution, all without any operational model in place says to me that what we are actually looking for is a demonstration of provability/suitability to ratify the design of a proposed solution.

Yep. I like to have real data to work with when designing a system. I would love to have a list, even a small list of maps that follow these rules:
1. A popular/semi-popular map.
2. Can be broken by a common setting being off.
3. Can likewise be fixed by toggling said setting.
4. Can be proven to be broken/fixed in most of the common source ports by toggling the setting.

Having even a small list, even a human-readable list would really make this thing tangible. Sure, you can fake some data, but being able to *manually* demonstrate that toggling a specific setting, in all relevant ports, makes the difference between a fun map and a train wreck. That's the first step. For this first set, it doesn't have to be machine-readable. Such a list would be useful, allowing the player to manually set the appropriate settings when a map was to be played.

It would require testing the setting in all the ports just to see that it really is a good compatible setting. I believe that, a valuable spec will present itself once we know what settings are really important and useful. We will fit the spec around the requirements vs. trying to devise a one-size-fits-all plan.

Step 2 would involve getting the data in machine-readable form, as appropriate, based on what we find is necessary, but that time is not yet here. I think discussions about optimizing size and what not are especially unproductive at this stage - it clouds the issue, and may not even be an issue.

Of course, none of this will work without a collaborative effort. There's no way for 1 or 2 people to properly index 1,000,000+ maps. But, remember, a great deal of maps will require no entry (or a blank entry, which signifies that this map works reqardless of settings, while implying that someone has confirmed this fact.)

Anyone care to take a shot at finding such maps?

[Gez]

How about this small list to begin with? Granted, you can ignore large parts of it which deal with ZDoom-specific stuff that are highly unlikely to affect any other port; but it's a start.

[kb1]

Gez said:

How about this small list to begin with? Granted, you can ignore large parts of it which deal with ZDoom-specific stuff that are highly unlikely to affect any other port; but it's a start.

Holy crap, that's exactly what is needed! Go ZDoom! I knew about that file before, but just didn't make the mental comparison.

It is a short list, but it has a wide variety of different types of examples - settings, map tweaks, etc. ZDoom is known to natively fix other things that some other source ports don't, so, it's not exactly definitive across the board, but, it's a great start. Is that an MD5 hash? (I can check the sources). I see some actual in-memory map hacks in there, which are actually in scope for my idea of what the compatibility database should be trying to accomplish. In fact, I always wanted a way to disable things that break coop gameplay, like the infamous "Walk into a room, door shuts behind you, monsters jump out and kill you, respawn, no way back into the room." Happens often with SP maps marked as "the starts are there, didn't test".

Of course, in-memory map hacks should be disabled for demo compatibility, unless the source port can save the map hack commands in the demo as well. That kinda opens up a new can of worms, if you're not careful.

[Gez]

kb1 said:

Is that an MD5 hash?

Yup.

[Quasar]

What are the ZDoom MD5 hashes computed from, btw?

Not only is the algorithm important (if ZDoom's MD5 algorithm had any bugs, they'd have to be adopted verbatim in order to get a matching hash), but also the exact input.

In EE's directory hack application code, only a wad file's header and directory are used to compute a SHA-1 hash. This data is already read out during the course of W_AddFile and so it was easy and efficient to base the hash on it, rather than reading the entire file.

[Graf Zahl]

Quasar said:

What are the ZDoom MD5 hashes computed from, btw?

All editable map lumps (label (for Fragglescript), things, vertexes, linedefs, sidedefs, sectors and behavior)

Check p_setup.cpp, MapData::GetCheckSum .

[kb1]

I'm thinking a compatible extension of ZDoom's compatibility.txt would make a nice fit.

I would consider adding a wad file hash too - might be useful if this things grows huge, but, maybe not necessary. I might also be inclined to include a size, which was a sum of the map's lump's sizes, just to add another level of insurance that the hash was unique to that map, but, it would already be astronomically rare to have a collision.

But, all in all, I find compatibility.txt an interesting eye-opener. Not only does it have non-Boom-specific options fixes, but also inline map tweaking, and core algorithm alterations (the sprite sort order stuff). It's simple, it supports inline comments, it's human-readable, it sets only the options required to get the job done.

I haven't checked the ZDoom source yet, so I run the risk of asking questions that I could find out on my own, but here goes:

1. ZDoom: Can ZDoom be made to (or already does) gracefully ignore options it doesn't understand?

2. ZDoom: Is this truly an external file that is read on startup, or is it currently compiled into the source?

3. ZDoom: Can ZDoom be coersed into reading a file other than "compatibility.txt", say, with a command-line option?

4. Other Port Developers: Any arguments for going forward with a possibly-extended-but-hopefully-compatible version of "compatibility.txt"? For me, it appears to cover my needs rather nicely.

[Graf Zahl]

kb1 said:

1. ZDoom: Can ZDoom be made to (or already does) gracefully ignore options it doesn't understand?

Not yet but no problem.

kb1 said:

2. ZDoom: Is this truly an external file that is read on startup, or is it currently compiled into the source?

It's part of zdoom.pk3, not an external file (in other words, it's an integral part of the ZDoom distribution)

kb1 said:

3. ZDoom: Can ZDoom be coersed into reading a file other than "compatibility.txt", say, with a command-line option?

Due to the above at the moment it can't be overridden by another file with a different name.

[Gez]

Graf Zahl said:

Due to the above at the moment it can't be overridden by another file with a different name.

Actually, there's no special check in place to make sure it comes from zdoom.pk3; however it has to be named "compatibility.txt" so it can only be provided by a stand-alone file or from a pk3 or similar archive as a wad file couldn't provide the full name.

[Quasar]

I don't see how wad files would even have to be a part of the conversation as they're too primitive. The file could be provided raw, as an optional drop-in - for example Eternity could allow it to be placed under the base folder. Also, EE will support zip-format archives pretty soon, now that zlib has been integrated successfully.

[kb1]

Cool, sounds like we may be moving towards a collaboration. Still like to hear from wesleyjohnson and DaniJ, and, hopefully some other devs.

[DaniJ]

ZDoom's internal compatibility.txt is effectively the definition of the verbose data collection format we have already discussed. The existence of it (and of which I was already aware, btw) does not change my opinion that this form is ill-suited to scaling up for use in a central database.

[Graf Zahl]

And why?

If you want something even more primitive I don't see much use in it.

[DaniJ]

I'm not arguing for a reduction in functionality. The issue is that of scalability and how this will be stored in and served from the central database. At present this file contains only a few dozen exceptions to a handful of maps that have had issues reported directly to the ZDoom devs. If we achieve the goal of a common, open central database used by all source ports, that users can actively contribute to - we should expect to see that the size of this dataset will (hopefully) grow pretty large given the sheer number of maps out there.

[kb1]

DaniJ said:

...we should expect to see that the size of this dataset will (hopefully) grow pretty large given the sheer number of maps out there.

So what? How large is large? It a large thing that we're trying to do. It takes whatever it takes. Hell, I don't care if it includes streaming video of the guy manually clicking the option settings, if it accomplishes the goal. I would pay to get such a file if it satisfies the need.

Besides, as you say, it's currently very small. We can worry about the size when we have the data - right now, it's a non-issue.

I'm not sure because I've extensively pruned my files, but I count about 100,000 wads. Let's make it 200,000. Let's say that 10% of them need any fix at all. With an average of 10 maps a piece. That's 200,000 entries X say, 100 bytes. That's 20Mb. A mapset is 20 meg. And, that's only necessary after indexing 2,000,000 maps.

[DaniJ]

So how many maps are there on /idgames? I think you are underestimating the cost in bandwidth to serve that as a chunk of data. It would be constantly changing and users will want the latest version.

[Gez]

A "constantly changing" thing, especially a collaboratively made one, would be on some version control system (SVN, GIT, etc.). There could be a monthly snapshot for most people, and those who want the very latest version would only download the update as handled by the SVN/whatever client.

[DaniJ]

That is hardly a convenient user interface.

[kb1]

DaniJ said:

I think you are underestimating the cost in bandwidth to serve that as a chunk of data.

kb1 said:

I would pay to get such a file if it satisfies the need.

Is this really what it has come to? File size? Are you kidding me? Finally, after all these posts and ideas, a system appears that does just about exactly what is required, is compact (it's pretty darn small), is already in use (proven) to small degree, easy to comprehend (human or computer), and we're down to file size? What's it gonna take? Cash? How many tens of people will download it? How often? Can you even answer these questions? What will they be downloading? Apparently nothing, because this file that doesn't yet exist is too big. But, how can it be too big if it doesn't exist? See what I mean?

If we start with compatibility.txt:
73 maps, 4,638 bytes: ~64 bytes/map, including comments AND 32 byte hash.

1,000,000 maps = 64Mb. Geez, what more do you want?

Edit:

Gez said:

A "constantly changing" thing, especially a collaboratively made one, would be on some version control system (SVN, GIT, etc.).

I think this is a good idea - if nothing else, it could prevent someone from f'ing the database up.

[DaniJ]

No its not just about cost. Using a hash has many other reasons to verify it than simply file size. Source ports need a way to efficiently store and search within the dataset. Not to mention it actually has to be updated. So are you suggesting we use a flat file database on the server (again)?

[Xaser]

Is someone going to suggest an alternative eventually instead of just saying "that's not going to work"? Unless the answer is buried somewhere within the last five pages...

My two cents from having just read this one page and coming up with some scenarios off the top of my head: An extended version of ZDoom's compatibility.txt (e.g. one large text file) seems like it might get large after a while, and having to either maintain a copy on the user's hard drive or read the whole file from the server each time sounds like it'd be a pain. The style of compatibility.txt (e.g. hashes based on each map) is just fine, but I'm thinking that if there's going to be a central database, why not use a proper DBMS?

The contents of compatibility.txt can be recreated in a relational database in a manner of minutes anyway, and something like that seems "easy enough" to go with. Each map has a hash, each hash value corresponds to a number of compatibility options, and a single SELECT statement is all it takes to get it. Beats having to parse a large text file every time, and going the route of having a different text file for each map/hash just screams "use a DBMS" anyway. :P

If we're talking about storing this "database" (and I use the word loosely now) locally, then as an end-user, I'm not sure if I'm going to bother with maintaining said hypothetical 64MB file aside from downloading it once and leaving it alone. But that's no real big deal, I guess, as long as each port gives me the option to give a path to the compatibility.db file instead of making me drop a copy of it in every directory ever. :P

To me, this is the sort of thing that makes more sense to have on a single, centralized server. But then it boils down to who's gonna run it.