Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Quasar

Simple linear format

Recommended Posts

Phobus said:

From a player/modder perspective I am, frankly, amazed that people try to run ZDoom mods in Doomsday (or vice-versa). Surely the reason we include text files in our downloads is so that people can identify what source port to run the mod with? You shouldn't defend that action, you should just tell people submitting these "bugs" that they're trying to run a mod in the wrong source port and can't expect it to work. Best case scenario for users would be for the port to be able to come up with an error message recommending the correct port for the mod, but that honestly sounds impossible to do accurately.

Before we implemented Hexen map format, we had frequent trouble with people who had run the ZDoom map converter utility on their IWADs for whatever silly reason ;) Just an example of the bizarre almost unanticipatable things end users will try.

Also now that EE runs a VERY narrow subset of ZDoom wads (2 or 3 of them total), trying previously ZDoom-exclusive mods in EE is certainly not a completely ridiculous thing to do ;) (This has been the case since DavidPH finished the new ACS interpreter).

Share this post


Link to post
Quasar said:

Before we implemented Hexen map format, we had frequent trouble with people who had run the ZDoom map converter utility on their IWADs for whatever silly reason ;)

Deepsea was probably the whatever silly in question, since apparently it only lets you create maps in the same format as the IWAD. (WTF?) Yay ill-designed tools.

Share this post


Link to post
Gez said:

Yay ill-designed tools.

This has probably done far more damage than any "ZDoom-ism." :P

Share this post


Link to post
Phobus said:

From a player/modder perspective I am, frankly, amazed that people try to run ZDoom mods in Doomsday (or vice-versa).

With some ports (including Doomsday) it is rarely necessary for the player to even look inside the Zip (let alone read the text file). The vast majority of mods can be loaded as-is with no need to unpack them first. Consequently this convenience factor and generally-works behavior tends to mask the compatibility complexities and users will come to expect it.

This is also the reason why any and all incompatibility issues should be caught and handled gracefully. You can't really purport convenience unless the user can also feel confident that any misstep they make will be caught by the application.

Logically one could argue that if you can't do both robustly that you should do neither...

Share this post


Link to post

I think it's a general attitude problem not to read the instructions - be it Doom mods or technical devices.

Sadly it's exactly these people complaining the most - even though it's their own stupidity causing the problems.


Concerning catching all potential issues, it's often easier said than done. Of course you can protect against most known problems but sometimes stuff comes along that just goes beyond all expectations.

Some years ago some joker uploaded a WAD to /idgames that contained an intentionally f*cked up level that bypassed all sanity checks in ZDoom and then later crashed in some truly inecplicable fashion. Not that it was designed to crash but the data in there made ZDoom go haywire in a completely unforeseen way by uncontrolled creation of some thinker structures. The data just happened to make just enough sense for the engine not to abort prematurely.

Share this post


Link to post

As far as I'm concerned, bar for truly exceptional cases such as deliberately contrived/broken data, that the application should not require the user to first read the instructions. In fact, to a large degree I believe that the fundamental concept of usage instructions to be draconian and outdated.

I am of the school that believes technology should be designed for the average person. If the average person does not read instructions then this should mandate the way technology is designed.

Naturally this is an ideal and certainly easier said than done, however I believe that we've reached a point where it is no longer acceptable to place such demands on users.

Share this post


Link to post

That may be. I don't like things that can't be used intuitively either and I'd consider both hardware and software that can't be used without reading the manual flawed.

I have more problems with users who refuse to read the manual *AFTER* encountering a problem. That's just utterly stupid.

But in case of downloading some add-ons, I'd expect people to check if the add-on works with the software they use.

Share this post


Link to post

I thought my posts were getting to long, so I try to compress, but some of those responses just leave me incredulous as to the level of misinterpretation that is possible. You might even lead me to start calling others silly, idiotic, too, and I want to resist any such crass impulse, so I don't. Read-the-instructions assumes the user has the information (often missing), and the author knew (see CommunityChest wads with zdoom levels), and the author documented it clearly (ha ha).

Of course I know that an editor generates such files, with whatever file identification that is appropriate. With text editors that can output in RTF, ASCII, and WORD format, it seems too obvious that the port specific editor can generate a selected output format. Are ZDoom wads actually created by wad editors with no Zdoom specific enhancements ???
I find that hard to believe.

A user could submit a text file as a wad, but at least it would be missing the "PWAD" header.

Versioning of file formats is necessary to prevent misapplication in any context where a user can submit a file of their choice. Quality tools do not leave it up the user to guard against submitting a file of the wrong version. Lower-quality tools have instructions that blame the users, mostly to avoid the work.

I do not want DoomLegacy to be a lower-quality port, so I do not accept the blame-the-user excuse when encountering wad enhancements that do not have version hints. I did not expect to play them all.
It would be best of all if I did not need to modify DoomLegacy because of ZDoom specific enhancements, but that is not possible anymore. It changed the kind of mistakes that users can now make.

Adding section names (lumps, textures, etc.), within the outer format relied upon the previous ports tendency to ignore such unknown sections. This has been the traditional wad format enhancement because vanilla doom ignored the unknown.

I had not said all this before because it just seemed too obvious to experienced developers. I can only suppose that the intention was to ridicule as much as possible by ignoring the obvious.

I think the only way this would be understood would be if another port accepted zdoom wads but with different parameter ranges (adapted to their existing code) and then their wads got confused with the pure zdoom ones. We would then see how much YOU like having to search the wad text file for hints, assuming the author left any.
And Community Chest projects would now have the author who did not realize they were using an incompatible alternate-zdoom-ism, instead of the pure-zdoom-isms.

As an example of proper headers: DoomLegacy save game and demo headers now include a unique DoomLegacy id line, the version of DoomLegacy, the version number of the format, the wad name, and more. Other ports can detect it immediately, and it is at least possible to identify all loading incompatibilities, without guessing. This is not to say that handling all previous save games or demo formats would be done, but it would be much easier. Just look at what prboom did along this same line, and then keep telling me that is all wrong, it should just be in the instructions.

Share this post


Link to post

What do you expect?

Doom originally and the WAD format in particular have absolutely nothing in place for any safeguards.

Your 'solution' is dead wrong though. It's never a good idea to cook up new formats or incompatible variations of a format to handle engine variations. It should be done within the boundaries of the format.

The only mechanism I could imagine would be to add a manifest file that contains information about feature and port requirements. It would also make it a lot easier to auto-configure the engine for old mods. Of course it would also have required a lot of self-discipline of *ALL* port authors - but the sad fact is that *NONE* ever bothered with it.

But that should have been done 14 years ago when Boom was released. Now it's too late because over the last 14 years a lot of stuff has been released that can contain anything.



Anything else would simply exceed the meaning of 'Resource container file'. Because that's all WAD is. It contains resources, without any semantics attached to it.

Share this post


Link to post
wesleyjohnson said:

It would be best of all if I did not need to modify DoomLegacy because of ZDoom specific enhancements, but that is not possible anymore. It changed the kind of mistakes that users can now make.

Example? Alternative graphics and sound formats can be taken care of with sanity checks (which given that most formats have a fourcc, isn't all that hard). Script additions can be taken care of by a proper parser which errors on unknown tokens (you'll want to do that anyways since a typo shouldn't cause a crash). I can't really think of anything else ZDoom does.

It is interesting since you start off asking if ZDoom wads can be created with wad editors without ZDoom enhancements (which is a yes btw, in fact you can get away with using operating system tools if you really wanted to), but at the same time your implying that editors should be designed to prevent user mistakes (such as putting a BMP into a wad).

To me ZDoom seems to be the design you're aiming for: "Give me whatever and I'll see if I can run it if not I'll tell you that I can't." vs "Give me a file I can understand otherwise undefined behavior will occur." The only thing I think we disagree on is if extensions should be presented as optional data vs required data. There is a lot more "required" things in ZDoom then there probably needs to be, but that's only because Raven didn't give Hexen's scripts regular syntax that can be ignored. If ZDoom should extend these scripts is debatable, but it's only a minor issue since it should only cause an error message for other ports.

Share this post


Link to post
wesleyjohnson said:

Of course I know that an editor generates such files, with whatever file identification that is appropriate. With text editors that can output in RTF, ASCII, and WORD format, it seems too obvious that the port specific editor can generate a selected output format.

Again, you're offshoring the work needed to create said different output formats to the tool authors. Many, many text editors are totally unable to handle any format other than simple ASCII text, by the way. Open some RTF or Word document in Notepad...

So not only are the port authors supposed to work on creating a new format, but they also have to convince tool authors to embrace it as well. It is unrealistic; especially when many, many of these tools are no longer maintained. Who would add the new container format to DEUTEX? Or XWE?

What happened that was closest to that was using zip files. And they were adopted because there were zip utilities already; so people could use them instead of DEUTEX or XWE to build up their mods.

wesleyjohnson said:

Are ZDoom wads actually created by wad editors with no Zdoom specific enhancements ???
I find that hard to believe.

Believe it or not, it's true. Eriance famously uses WinTex, for example.

wesleyjohnson said:

A user could submit a text file as a wad, but at least it would be missing the "PWAD" header.

If the header is all you're using, then any text file which begins with PWAD will be misinterpreted as a wad... You have to perform sanity checks on file size, number of entries, offset values, etc.

wesleyjohnson said:

Adding section names (lumps, textures, etc.), within the outer format relied upon the previous ports tendency to ignore such unknown sections. This has been the traditional wad format enhancement because vanilla doom ignored the unknown.

.... Aaaand also because it doesn't require modifying the container format; so any existing tool can still work.

wesleyjohnson said:

As an example of proper headers: DoomLegacy save game and demo headers

Not comparable. Saves and demos are both written and read by the same engine. Okay, you do lose compatibility with savegame editors and demo editors; but those tools only appeal to very small niches of players, so it's not really a problem. FYI, ZDoom saves and demos are versioned too.

You really cannot compare savegames (normally used only by the same person who made them, and only used in a relatively small time scale since you're not likely to load up a save you made three years ago) or demos (which sites such as the DSDA only host if they have the required text file with precise engine information anyway) with mods.

Share this post


Link to post
Graf Zahl said:

The only mechanism I could imagine would be to add a manifest file that contains information about feature and port requirements. It would also make it a lot easier to auto-configure the engine for old mods. Of course it would also have required a lot of self-discipline of *ALL* port authors - but the sad fact is that *NONE* ever bothered with it.

Actually the Doomsday frontend introduced add-on manifests back in 2005.

Anything else would simply exceed the meaning of 'Resource container file'. Because that's all WAD is. It contains resources, without any semantics attached to it.

That is a stupid argument Graf, of course it can contain semantic usage information without "sullying" the container concept. Its certainly a far better idea to include manifest -like data as a file inside that container than simply assuming every tool which ever needs to open that container automatically understands everything.

Share this post


Link to post
DaniJ said:

Actually the Doomsday frontend introduced add-on manifests back in 2005.


Even that would have been too late.

90% of the perceived 'damage' was done long, long before that.
And I do not even consider the lump format issues the biggest problem here but the fact that an engine cannot auto-configure to existing WADs due to lack of information what's needed.

The first moment incompatibilities were introduced this should have been done. But the awareness of this issue didn't exist back then.

DaniJ said:

That is a stupid argument Graf, of course it can contain semantic usage information without "sullying" the container concept. Its certainly a far better idea to include manifest -like data as a file inside that container than simply assuming every tool which ever needs to open that container automatically understands everything.


The major flaw in here is to assume that an engine has to 'automatically understand everything'. That's not the case, not even close! Of course older engines without any sanity checks will run into problems, but these days I wouldn't ever think of loading some data without checking its validity. If I don't know what's up, I'd print an error and abort.

I'd never, ever consider it good design to start the engine despite running into some obvious problems during initialization.

So:
- if I find a supposed graphic lump that can't be validated, print an error.
- if I find a sound lump that can't be validated, print an error.
- if some definition lump contains unknown syntax, print an error.
- if you perceive an inconsistency, print an error.
- if anything else fails - guess what: print an error!

If all this is done there shouldn't be any problems anymore - unless of course you insist on keeping the engine run with this mod, despite knowing that it would not work.

If you do not want errors I don't see any justification to complain that other engines allow data you can't use. In that case you have to be prepared for that kind of data. And that's your task, not mine!

Share this post


Link to post

The problem of manifest or other metadata is, again, that they'd have to be created either by the WAD management tools themselves automatically (and then any WAD created by an older tool without a manifest generator is manifestless); or by the modders themselves (and then you have to pray that they will bother making it).

Having the manifest lump necessary for the port to accept running it would be the only way to force modders to put them in their mods; however it also means that your port is no longer able to run any of the existing mods... It's a suicidal step that the port would be making.


So, these are the three pitfalls that await any attempt at having the WAD pre-digested for the port:

  • Absolutely no tool support.
  • Modders will not bother unless it's enforced.
  • Backward compatibility is broken unless not enforced.
This is true of any suggestion here, be it a slightly different container format or the addition of some new MANIFEST, METADATA, or PWADINFO lump. Even if you do implement it, you cannot expect it to be available.

There is actually another pitfall: what if it's bogus? For example, a lot of people are still convinced that the Doom picture format is BMP, and that index 247 is both cyan and transparent. These misconceptions arose from how certain tools hid the conversion operations from them, letting them assume there was no conversion at all. To give another example, Doomworld's very own music recording are provided with an MP2 extension; yet they are not actually in MP2 format; they're MP3 inside a RIFF-WAV container.

What do you do then, when the manifest gives you incorrect or misleading metadata? It's just as good as not having a manifest at all. In fact, all it does it open a possible attack vector for trollwads if you think the presence of a manifest means you don't have to bother sanity-checking and validating the data... Because be it malice or incompetence, or even the odd freak case of data corruption (XWE, I'm looking at you), you always run the risk of running into nonsensical data. Doom is prone to that even with valid data.

At most, the manifest can be used as a hint for the rare cases when a particular data sequence looks like it could be several different types of formats. For example, a raw graphics with a size of 4096 bytes could be 64x64, but it might also be 32x128, or 128x32, or 16x256, etc. You'll still have to check that the size is indeed 4096 bytes; but you'll require metadata to give you exact dimensions. A concrete example would be the Jaguar Doom IWAD textures; they are stored in a raw format with the first 320 bytes repeated at the end. Because of that, you have to get the dimensions from the TEXTURE1 lump.

Share this post


Link to post
Graf Zahl said:

The major flaw in here is to assume that an engine has to 'automatically understand everything'. That's not the case, not even close! Of course older engines without any sanity checks will run into problems...

Exactly. Hence why it is a logical error to mix data formats if validity cannot be unambiguously determined. The Flats "namespace" in WAD is one such example - it should have been left alone.

But anyway, all these circular arguments achieve nothing. The reality is that these mistakes have been made and active ports must now deal with the fallout.

Not learning from these past mistakes is another matter entirely, however.

Gez said:

The problem of manifest or other metadata is, again, that they'd have to be created either by the WAD management tools themselves automatically (and then any WAD created by an older tool without a manifest generator is manifestless); or by the modders themselves (and then you have to pray that they will bother making it).

Having the manifest lump necessary for the port to accept running it would be the only way to force modders to put them in their mods; however it also means that your port is no longer able to run any of the existing mods... It's a suicidal step that the port would be making.

A manifest which requires special tools to author would be yet another design false step. The only logical format for a manifest is a text file (with markup of your choosing).

Requiring a manifest is not suicide at all. The whole point in the manifest is to allow an application to load data it otherwise cannot. So the use of manifests is a gain and certainly not a loss.

Share this post


Link to post
DaniJ said:

Exactly. Hence why it is a logical error to mix data formats if validity cannot be unambiguously determined. The Flats "namespace" in WAD is one such example - it should have been left alone.


Strangely enough ZDoom's texture loader has absolutely no problems with this. On the contrary: Before I implemented a sane texture format checker it was causing a lot more issues than it is now.

DaniJ said:

Not learning from these past mistakes is another matter entirely, however.


For Doom it's too late.
There haven't been any major changes in recent years and I don't expect any new to come - with the sole exception of FMod adding support for new sound formats, which then would be supported by ZDoom automatically.

DaniJ said:

Requiring a manifest is not suicide at all. The whole point in the manifest is to allow an application to load data it otherwise cannot. So the use of manifests is a gain and certainly not a loss.


Depends on what you can enable with it.
Assing a manifest to unblock PNG support for example is a ludicrous proposition in my book.

What it *IS* good for, though, is to declare the engine features it wants to use. But of course this requires a lot of care how new features get implemented.

Share this post


Link to post
Graf Zahl said:

Strangely enough ZDoom's texture loader has absolutely no problems with this. On the contrary: Before I implemented a sane texture format checker it was causing a lot more issues than it is now.

Why do you feel the need to continuously rebut a logical argument with "ZDoom can handle it so..."? Whether it can or not is completely irrelevant - you caused the problem in the first place, no amount of posturing can make up for that.

DaniJ said:

Not learning from these past mistakes is another matter entirely, however.

For Doom it's too late.
There haven't been any major changes in recent years and I don't expect any new to come - with the sole exception of FMod adding support for new sound formats, which then would be supported by ZDoom automatically.

Er, what? How does any of that relate to what I said? You aren't seriously trying to argue that a previous mistake means there is no need to do the job properly in future?

Depends on what you can enable with it.
Assing a manifest to unblock PNG support for example is a ludicrous proposition in my book.

Well duh.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×