AnalyzeWad and WA file

Here is an idea (an aside from the discussion in another thread), that does not require any cooperation from the Zdoom project, or any other port author.

Someone was collecting information on differences between ports.
Let's create a program (call it analyzewad) that looks at a wad and figures out its properties, what port may be needed to run it, and what settings may be required.
It will be run once on any wad, so it can afford to take the time, and double check between linedefs and sectors, and look into things like texture formats, and PNG.
So when a problem wad appears, the user could run anaylzewad on it, and would be informed of the properties, and thus could deduce the causes of the problem.

One option of analyzewad program would be to write a file (let us give it an extension of ".wa" for wad analysis, and let us call it a WA file).
It would be in text format, and can be edited afterwards for customization. The WA file would have the same name as the wad, with ".wa" extension instead, so the two stay together.
Distributed zip files could include a WA file themselves.

The one difference from the previous discussions of database would be that the individual WA files could be manipulated easily and multiple versions could exist for a wad (for game setups ??).

The WA file would point to everything that is needed to play the wad.
The format and content of the file would determined by the analyzewad program, up until a competitor program appears. But still would be
much easier to define (than other proposals), and any port could from load it (fully or partially).

A fully conforming port could be presented with only the WA file, and would load everything that it indicates. It would still be allowed to use other command line switches to load other PWAD files in addition, if the user so wanted to. Interaction with conflicting switches would be up to the port.

Analyzewad can only supply a fraction of the lines defined below, the others are for hand customization. They serve much of the same purpose as other wad descriptions that have been discussed.
As always, any line or flag that is not recognized by a port is expected to be ignored.

By copying a WA file and customizing it, a user can keep a special play setup and use it easily.

Proposed Content of WA file:

# WA file header with version number
WA:V01
# This is the main wad, load it first
WAD: ugly.wad
# This is the game required
# (from list DOOM1, ULTDOOM, DOOM2, PLUTONIA, TNT, HERETIC, HEXEN,
# others added as needed or requested.
GAME: DOOM2
# header starts as Port: ALL
# flags describing discovered facts about the wad
# additional flag defines do not change the version number
HAS_REQ_PNG
HAS_ZDOOM_xxx
HAS_ZDOOM_yyy
HAS_LEGACY_L301
HAS_LEGACY_L302
HAS_EDGE_L302
HAS_EDGE_BARREL
# load these pwad too, after the main wad
LOAD: gothic2.wad
LOAD: simpsons.wad

# PORT specific section, (using Legacy as an example port name)
# if a port does not find its name, it could use the PORT settings
# of a close relative port.

PORT: Legacy V1.44
# everything in here is only loaded by the named port version.
# how the port version is named is up to the port.

PORT: Legacy
# everything in here is only loaded by the named port, upto the next port line
# All versions
# Load the matching PORT section without version, even if a PORT section with version has also been loaded

PORT: Legacy DEFAULT
# load this only if no other PORT section for Legacy has been loaded.
# all version specific PORT sections for Legacy need to precede it

PORT: ALL
# everything in here is loaded by all ports, even if another PORT section has been loaded.

PORT: DEFAULT
# load this only if no other PORT section has been loaded.

Port specific suggestions:

SET: <keyword> <setting>
# These would be port settings. They are port specific keywords.

SET?: <keyword> <setting>
# An optional port setting, should ask the user before executing the
# setting.

#SET?: <keyword> <setting>
# a disabled setting


A launcher could also be created (probably system dependent).
It would read the WA file and try to launch the first port mentioned that is present on the system, (or interactively, present the user with a list of ports to select from),
and launch that port using command line switches to select the game, load the main wad, load the additional pwad, and select the options.
This would generically cover any port that did not implement any WA support.
The launcher could use a user default WA file to enable settings that
the user prefers.

SET-: <keyword> <setting> # user default
# used in absence of other WA setting for the <keyword>

SET!: <keyword> <setting> # user override of WA file for <keyword>

OPTION: <keyword> <setting>
# a command line setting, exactly as it appears on the command line
Example:
OPTION: "-opengl"

# This ability could be part of SET, if a way to denote which keywords could be uses on the command line was invented.
Maybe:
SET-: <keyword> >> <command-line-text>
Example:
SET-: graphicsmode >> "-opengl"

This is first cut at the idea. Please improve on the idea, mention problems, or start another thread if you want to be tiresome.
This is one idea that does not require everyone's cooperation.

I whole heartedly approve of the concept.

This is the sort of thing that textual manifests are born to do. Doomsday (and the Doomsday Frontend) already has an established set of features and file formats for this very purpose. It seems to me that it would be better to build on this functionality rather than starting again.

I fully support stealing any or all of this idea into any other similar project. I could not do anything on it for 6 months at least.

The one difference is that this way is a smaller investment in time to get it started. First target would be Zdoom wads because of the payoff.

One advantage is that it does not need a big database, but could be partners with a database too. A database could load and output individual WA files.
Database could present WA format data to ports (thus avoiding a complicated interface) and such a method would make it easy for ports to load custom WA files too.

Database is not easily edited to create custom play setups like the WA file could.

Have not seen DoomsDay stuff. Does that work on Linux?
How close are its formats to the capability offered by WA files ?

wesleyjohnson said:

One option of analyzewad program would be to write a file (let us give it an extension of ".wa" for wad analysis, and let us call it a WA file).
It would be in text format, and can be edited afterwards for customization.

Tsk tsk, if it's not XML-based... ;-)

Sorry, looked at Doomsday, but you are going have to tell me how that would help me screen and adapt wads for playing on Legacy.

Of course, I approve. Been trying to do this myself algorithmically for a while now.

wesleyjohnson said:

Have not seen DoomsDay stuff. Does that work on Linux?

Yes.

How close are its formats to the capability offered by WA files ?

Pretty close. See the Snowberry User Manual for more info (particularly the section entitled "Advanced Guide"). Note that some of this functionality has already been imported into Doomsday also (we plan to drop the frontend entirely in future).

I am asking myself why I would need such a program.

It certainly makes a wonderful exercise for programmers to strut their stuff, but, for me as a layman,
it is a lot more reliable to quickly load a wad into Slade3 and see exactly what the wad is all about.

But then, I may misunderstand the purpose of the program.

Kappes Buur said:

I am asking myself why I would need such a program.

Yeah, me, too.

Kappes Buur said:
It certainly makes a wonderful exercise for programmers to strut their stuff,
[/B]

Correct. That's probably the only thing it's useful for.

Kappes Buur said:
but, for me as a layman, it is a lot more reliable to quickly load a wad into Slade3 and see exactly what the wad is all about.
[/B]

I have to agree. A tool like this only makes sense if we can assume that WADs come without text files - or that the users are too dense to open the text file and read it - and instead prefer to waste even more time to run a clunky tool to see what it is all about.
Store the information to use - by what?
Some database tool that tells the user what port the WAD is for? Sounds useless.
Use it by a launcher to auto-decide what port to run the WAD with? Nah, I want freedom of choice.
Use by a port's WAD loader? Now it gets 'ugh'. A good error checker and data validator does the job 10 times as well and is a much better thing to spend all the time on that'd be needed to write such a tool.

Kappes Buur said:
But then, I may misunderstand the purpose of the program. [/B]

Read here:
http://www.doomworld.com/vb/source-ports/61636-simple-linear-format/

Look specifically for the posts by DaniJ and wesleyjohnson.
I have to admit that even after all this discussion I fail to see the point of all this.

One use I could see for this (provided it works satisfactorily) is to make an idgames bot that would analyze the entirety of a local copy of the archive, and update the database with new filters. For instance, there is no search for "required ports". Try to search for Eternity mods on the Doomworld/idgames archive frontend; you'll have no other choices than "whole textfile" and that gets you tons of false positive because Eternity is a common name. (Same with Doomsday.)

So for automated analysis of thousands of files, it makes sense.

Architecturally this model would greatly improve robustness and cross port mod compatibility. This is not something that the user would have to be involved in, it would happen automatically in the background.

Imagine if you will a hypothetical library which any port could link to. This library's sole purpose is to analyse a set of lumps to determine which port's features are employed. This information would then be written to a manifest with appropriate hashes for the source data.

A sourceport would link to said library and cache the manifests it produces for the lumps given to it to analyse. Upon loading a mod (or map or whatever) the manifests could be used to resolve ambiguous cases where two implementations differ and which cannot be determined by looking solely at a subset of the data.

Take for example the data placed into the map marker lumps by various different source ports. Often it is not possible to determine which port the map is designed for (and thus the way it should be interpreted) through just this lump alone. In this situation the whole-mod analyses performed by this library could be used to help resolve the situation.

Not only does this improve robustness but also moves much of the overhead involved in validating data and handling those cases where it does not out of the engine itself.

Its a complete top down change in approach - don't attempt to load what we don't know the format of, rather than the current approach of trying to load everything and then contain the issues which occur due to this naive approach later.

I think a user would far prefer this model as it means they get a break down of exactly why their mod doesn't work in port X. Rather it appearing to work (because load doesn't fail) only to find out during play testing that it doesn't for unknown reasons which the modder then has to figure out for themselves.

In theory this sounds like an acceptable idea. In practice I foresee a veritable nightmare.

Here are some of my apprehensions:
Not that I am against what you propose, I just cannot see it ever reliably playing, for example
a pwad for EDGE or DOOMSDAY in ZDoom/GZDoom, or vice versa. If source ports were to use the same
lump structure, then there would be some hope for success. How well would the DECORATE lumps alone
fare in DOOMSDAY? Consensus between source port developers has never been a strong point. I am
still surprised that UDMF has been so readily agreed upon by so many developers.

Who is going to maintain such a database? If that is not always current then it is not useful.

Each source port is still in flux, so it will be quite some time before they are finalized.
How can this program assure complete compatability when a source port adds a new feature?

Most players are partisan (quite a few extremely so) to a certain source port, or on the outside maybe two.
I can see, especially, the Vanilla DOOM or Chocolate DOOM adherent never using this program.

I think it tries to do too much and is therefore inherently too complex. Routing all those different types of data to the various subsystems they need to reach would require thousands of lines of code, and most of it redundant to things we already have. In Eternity for example:

• GFS files are available for loading of multi-file mods.
  
• Response files are available when all you want to do is set command lines.
  
• Console scripts are for running console commands at startup.
  
• UDMF (when it's implemented) has its own version control for telling the port for which a particular map has been designed.
  

I'm really not prepared to add redundancy on top of all of those systems in what I feel like is a misguided attempt at a One Ring solution.

There is no need for a central database, that part of the proposal is entirely redundant.

Take that out of the equation and I don't see where the complication lies - its just a series of data conformity tests which we can already source from each port's existing validation logic.

This isn't strictly even about being able to load a mod from another port. Its about providing a better experience to the end user.

Note that this doesn't actually require any cross cooperation at all. An effective implementation could leverage an existing scripting language to allow anyone at any time to write new conformity tests.

It does not require a database, nor any cooperation between ports, except for what is the basic WA file format legality. A specific port can customize their port specific lines, and only needs to keep it legal so that other ports can read past it using the basic format rules.

A specific port would not have to route a whole lot of information.
What info a port it interested in is up to that port.

It got complicated because it is a perfect place to put some of the other wad information that has been talked about, like which switches to use to best play a wad.

Most ZDoom wads would not even have sections for any other port because those wads cannot be played reasonably. What was provided was the identification of the wad features so that it could be rejected before investing time in trying to make it work.

That it is better than getting out an editor and hand searching for clues should be obvious. And if one player did it already and included WA file for the wad, then why should most downloaders have to repeat a hand analysis by editor.