Simple linear format

[Blzut3]

Probably the most important aspect of designing engine features is that you don't want users to have to think. If a user wants to use a PNG for their flat the logical conclusion is to put the PNG between F_ markers. Doing anything else causes needless frustration, which you imply we should do since some ports don't want to bother with checking the validity of input. (Which by the way, pretty much the number one rule in software development is to NEVER trust user input.) Besides having high resolution/color depth as an optional replacement won't really help with compatibility since practically no one will bother with creating low resolution versions for cross port compatibility (if you force them you'll just end up with all black/transparent textures).

I am by no means saying that everything that ZDoom implements is done perfectly, but when it comes to supporting alternate image/sound formats the only way to do it in a user friendly way is allow the data to be inserted where it normally is and have the engine either accept or reject it. The average user does not care about the technical details.

[Quasar]

What occasionally gets to me is the one-sided nature of ZDoom driving the standards that get implemented. Since HIRESTEX is apparently a bad example somehow since it involves causing crashes that can be prevented through format checks, let's talk about Hexen data scripts.

It is impossible for a port to implement them now and ignore ZDoom extensions which can be placed into the files without any indicator of their presence. At least, your parser for them must be tolerant of those extensions or any map that uses them will unnecessarily barf out.

Granted this mistake has been acknowledged in the past and partially made up for with the suggested use of alternate lump names; unfortunately mods released last week continue to abuse the old lumps instead of using the alternatives. It's too little too late but the gesture is of course appreciated.

However, what's the chance that any other port implementing customizations to those same lumps would see the support added to ZDoom? Very little. It would be rejected with the suggestion that a different lump be used, even if the extensions are as small as a few new field types rather than actual syntactic differences. When I asked to have the $edfoverride keyword ignored in ZDoom it was kind of like I insulted Randy's mother :P

ZDoom is similarly free to exclusively define the ACS bytecode spec. Ranges of instructions are defined for the various ZDoom descendents (GZDoom, Skulltag, etc) but where for example is a block of free opcode indices that I could be assured would be safe to assign to Eternity-specific extensions? There's not one. That means I can't really add any without introducing a new bytecode container format (which is one reason that Eternity has gone to lengths with its bytecode translators - thanks DavidPH - to make that possible if it has to be in the future). Same deal with ACS thingtype #'s. I cannot assign any in EE that aren't assigned in ZDoom, even where I see that there are glaring omissions.

[Graf Zahl]

Quasar said:

What occasionally gets to me is the one-sided nature of ZDoom driving the standards that get implemented. Since HIRESTEX is apparently a bad example somehow since it involves causing crashes that can be prevented through format checks, let's talk about Hexen data scripts.

It is impossible for a port to implement them now and ignore ZDoom extensions which can be placed into the files without any indicator of their presence. At least, your parser for them must be tolerant of those extensions or any map that uses them will unnecessarily barf out.

Granted this mistake has been acknowledged in the past and partially made up for with the suggested use of alternate lump names; unfortunately mods released last week continue to abuse the old lumps instead of using the alternatives. It's too little too late but the gesture is of course appreciated.

That's always the problem with mistakes made long, long ago. Most of these problematic decisions were done more than 10 years ago when port interoperability was not a concern or simply not being thought of.

As you point out, such mistakes will haunt us to the end of days but so do some of the design mistakes in the original Doom. Once out these things cannot be contained anymore unless you break compatibility - which logically nobody wants.

However, what's the chance that any other port implementing customizations to those same lumps would see the support added to ZDoom? Very little. It would be rejected with the suggestion that a different lump be used, even if the extensions are as small as a few new field types rather than actual syntactic differences. When I asked to have the $edfoverride keyword ignored in ZDoom it was kind of like I insulted Randy's mother :P

It never hurts to ask. Although I almost stopped contributing to ZDoom due to lack of motivation. If some of these things can be ironed out I'll do it.

On the other hand, I would really prefer to deprecate these lumps altogether and create something better for the job - something that can be extended without causing problems to other engines (like ZMAPINFO for example - that was the direct result out of some problems related to MAPINFOs shortcomings)

ZDoom is similarly free to exclusively define the ACS bytecode spec. Ranges of instructions are defined for the various ZDoom descendents (GZDoom, Skulltag, etc) but where for example is a block of free opcode indices that I could be assured would be safe to assign to Eternity-specific extensions? There's not one.

Have you ever asked? If not, you got no right to complain. ;)
I think the biggest issue here is that Randy maintains ACC and adding new bytecodes requires changes to the compiler - which is why I added an external function definition interface some time ago to add new functionality without having to recompile the compiler each time.

That means I can't really add any without introducing a new bytecode container format (which is one reason that Eternity has gone to lengths with its bytecode translators - thanks DavidPH - to make that possible if it has to be in the future). Same deal with ACS thingtype #'s. I cannot assign any in EE that aren't assigned in ZDoom, even where I see that there are glaring omissions.

Again, if you don't say anything, nothing will happen.
If you see glaring omissions, point them out and we can certainly define them for all ports to use.

[Blzut3]

Quasar said:

However, what's the chance that any other port implementing customizations to those same lumps would see the support added to ZDoom? Very little. It would be rejected with the suggestion that a different lump be used, even if the extensions are as small as a few new field types rather than actual syntactic differences. When I asked to have the $edfoverride keyword ignored in ZDoom it was kind of like I insulted Randy's mother :P

This?

Quasar said:

ZDoom is similarly free to exclusively define the ACS bytecode spec. Ranges of instructions are defined for the various ZDoom descendents (GZDoom, Skulltag, etc) but where for example is a block of free opcode indices that I could be assured would be safe to assign to Eternity-specific extensions? There's not one. That means I can't really add any without introducing a new bytecode container format (which is one reason that Eternity has gone to lengths with its bytecode translators - thanks DavidPH - to make that possible if it has to be in the future). Same deal with ACS thingtype #'s. I cannot assign any in EE that aren't assigned in ZDoom, even where I see that there are glaring omissions.

I'm maintaining a comprehensive opcode/function list with in the ACC++ repository. If you need a new opcode please let me know. For the most part though the internal function call opcode Graf mentioned should cover most new opcode needs. For those just pick a constant for EE.

[Quasar]

If I can request a pleasantly large enough opcode block, say 1000 codes or so just for growing space, consider it requested ;) If that's too many, bump it down to whatever will work. EE supports all three bytecode formats now, so which one we're talking about is of course important, and I'm not sure which one I should be asking for myself ;)

[DaniJ]

@Graf and Blzut3

Of course by all means make no attempt to even understand my post or the issue you guys have caused. Why make even that much effort?

Graf Zahl said:

I think you fundamentally misunderstand this namespace. It is in no way meant to contain anything optional. Replacements from this section will be used unconditionally.

I think you fundamentally misunderstand the feature. Give me one good reason behind the design of the implementation given ZDoom already supports PNGs in the Flats namespace.

Re: ACS bytecode compatibility

And this is the reason why with Doomsday we plan to sidestep the whole sorry mess by decompiling the BEHAVIOR lumps and translating them to Doomsday script at runtime. If the bytecode won't decompile then the resultant script won't even be loaded.

[Quasar]

Blzut3 said:

This?

Now that I look at it again I see that my memory is distorted. I suppose I originally took Randy's comment as a bit backhanded, as it wasn't really relevant to the request, but, in retrospect I don't believe it was really meant that way. Or I hope at least.

Thanks for linking it and clearing it up.

[Gez]

Most of the mods created for ZDoom wouldn't work at all in other ports. It's not just a question of also adding PNG code for flats, there's a different map format, DECORATE actors, MAPINFO stuff, and so on. If you add PNG code just so that you can play ZDoom maps in some other ports, you'll have a ton of additional work to do... To be honest, unless you go the Vavoom route and just lift ZDoom code wholesale, it'll tireless work that will end up being mostly pointless.

So, what is affected is only ports that aim to be for "ZDoom and...", like many of Eternal's recent works (e.g., Voodoo Guns). Then it's the modder's responsibility to ensure that it works on all target ports.

This has led to requests for Randy to implement stuff in ZDoom that makes interoperability easier.

[DaniJ]

Its not about wanting to support ZDoom mods Gez. This is simply about not wanting my app to crash when users attempt to load ZDoom mods in it. This is a rather different notion IMO. Much like the ACS compatibility issues, the poor way in which this feature was realised makes even this much hard work for little gain. This is the epitome of careless design.

[Blzut3]

DaniJ said:

Of course by all means make no attempt to even understand my post or the issue you guys have caused. Why make even that much effort?

Mind re-explaining then with links to relevant information? When you say HIRESTEX I think of this. Since you mention namespaces I'm thinking of the HI_START/HI_END markers or the hires/ directory in ZDoom format pk3s. Your post seems to imply that you believe that alternate format images should be in there with a standard format image in the flats namespace. So what were you suggesting?

Quasar said:

If I can request a pleasantly large enough opcode block, say 1000 codes or so just for growing space, consider it requested ;) If that's too many, bump it down to whatever will work. EE supports all three bytecode formats now, so which one we're talking about is of course important, and I'm not sure which one I should be asking for myself ;)

While there's nothing wrong with reserving such block, I do have to ask what you plan on using them for though? At the moment there's about 360 opcodes where quite a few of them are built in function op codes (lets call them Hexen style functions to avoid confusion). If you're not implementing new language concepts, then the callfunc (opcode 351, ZDoom style or ACSF_) should work fine. This instruction provides information on the number of arguments so unknown functions can be ignored. For these you just need to give us a constant you want EE's functions to start at and we can reserve them for you. (ZDaemon's functions start at 19620 for example. I guess I would suggest 17733 for "EE".)

Because Hexen style functions are deprecated, I do kind of want to be conservative with opcodes at this point so I was referring to approaching me individually for each opcode (or set of defined opcodes). This way I can keep the manual up to date with the purpose and format of each code. But, if you're planning on implementing a new instruction set for ACS then sure I guess. Same thing would apply for ACSe/ACSE chunks.

On a side note, the byte code format is irrelevant for the discussion of op codes since the actual code is the same for each. Only the meta data format changes. (Although ACSe has compressed byte code.)

[kb1]

wesleyjohnson said:

I have in DoomLegacy some detection of special linedefs to differentiate Boom wads from DoomLegacy wads.

On that note, I have some questions for you and Quasar, about detecting the differences between DoomLegacy, and Eternity wads. It appears that Eternity portal linedefs use type numbers that conflict with Legacy 3d floor line types. Was that done intentionally by SoM?

I am probably wrong, but I got the impression that maybe he was distancing himself from his previous 3d floor efforts in Legacy. If that assumption is true, does that explain the conflicting linedef types?

More specifically, I have written some code as well, that tries to determine what port is needed to run a given map. This effort is hindered by these line type conflicts. Is there any known way (or just decent way) to definitely, algorithmically, choose the appropriate port? (Sorry if this is slightly off-topic, although it kinda is related in a way).

[kb1]

Graf Zahl said:

In other words: the first thing ZDoom ever did (implementing support for Hexen maps) was wrong.

Even Boom made significant changes to the way WADs are handled and WADs made with Boom in mind won't work on non-compliant engines.

The original WAD format was so broken that sticking to it would have meant not being able to advance the engine much.

Sorry, but you don't make any sense at all. With that attitude no progress would be possible.

Graf, that's bullshit and you know it - that's not what DaniJ means, and I *know* you're smarter than that. Yeah, DaniJ's post makes sense to me, and I'm sure it makes sense to lots of people - it's straight-forward, spelled correctly, uses good grammar, and summarized nicely. But most of all, he's right.

"The original WAD format was so broken"??? What?

Yes, Boom did make significant changes, and, yes, non-Boom engines will fail on Boom stuff. But the Boom approach is way different than the ZDoom approach:

* Original DOS editors practically already support Boom enhancements out of the box (a notable exception: SWITCHES/ANIMATED lumps - why they decided on these strange formats, I'll never know).
* Untested Boom support is added to a source port in a day.
* No external libraries are required to be linked, dynamically linked, etc for Boom support.
* Boom *did* "advance the engine much", and did it using the original WAD format.

And, one of my personal favorites, most Boom engines play original demos! Not only that, but they also play advanced Boom demos. Hell, PrBoom+ can play most all flavors of original Doom, Boom, MBF, etc. But, not ZDoom. Heh, ZDoom cannot even play most ZDoom demos...

Don't get me wrong, I understand that ZDoom's philosophy is different - basic support for most "easy-to-support" game formats, with a slant for improvement, bug-elimination, and advanced editing. Noble causes. That's all well and good.

But, can you take a guess as to why the "easy-to-support" formats are easy-to-support? That's right, it's because those authors gave a shit about compatibility, and shared features, and easy implementation, and openness, and humbleness, and adding features without sacrificing compatibility, at least to some degree. Here's the two mindsets. You guess which mindset goes with which port:

#1. "Let's add this cool new feature."

#2. "Let's think about how we could possibly add this cool new feature, it a way where it can either be seamlessly added to existing structures, using existing code. If that's not possible, let's do some research: Probably there are some other ports in existance that also want to add features. Maybe we can collaborate, and devise a solution that can be considered a standard, so that eventually all ports can be improved. If we can, great - let's publish that standard. If we cannot, we will try again. We will repeat this cycle until there's no chance for agreement. When we have exhausted all efforts in collaboration, we will create the new standard ourselves, but we will make damn sure that our new standard would allow the other ports to easily use our standard. Because we are considerate and believe in compatibility, we will use the most simple structures and code possible."

Somewhere in this thread, DaniJ, Quasar, and others state this point much more eloquently than I do - DaniJ: "...inconsiderate design" and Quasar: "...the one-sided nature of ZDoom driving the standards".

What I want to know is: Why would anyone, especially an accomplished programmer, defend this lazy disregard for map editors, port authors, tool authors, documentation writers, and to a lesser extent, players? It's utterly ridiculous, and it's self-perpetuating.

So, how do you improve on these points? First you have to want to. And that's what I think Quasar was doing when he created this thread.

Back to the original topic, if possible

Quasar:
With all the new talk these days about 32-bit rendering, I would love to see an image format that handled 8-bit as well as 32-bit images. It would be *really* nice if said method could optionally capture both the 8-bit and 32-bit versions of a patch/texture/flat/whatever, without the need for per-pixel closest-color matching, etc. That may not be what you were envisioning - so be it. But, I *do* worry that, when many ports start to adopt the 32-bit rendering stuff, we will have umpteen image formats to handle. Ironically, I'd almost be tempted to endorse PNGs, for the simple fact that, if we've all got to use *something*, I'd hate to hypocritically add another format for everyone to support :(

[DaniJ]

Blzut3 said:

Mind re-explaining then with links to relevant information? When you say HIRESTEX I think of this. Since you mention namespaces I'm thinking of the HI_START/HI_END markers or the hires/ directory in ZDoom format pk3s.

HI_START/HI_END is indeed what I was thinking of. Its not easy to keep track of ZDoom features what with the number of similar sounding names, overlaps and deprecations.

However, it really doesn't matter much. Like others here, I guess this was largely a general rant about the blasé attitude of the port and its authors. Not only do they poorly replicate existing functionality in other ports, they have the sheer audacity to then suggest that stuff like this is our problem because we don't do things the way they do.

[Graf Zahl]

DaniJ said:

they have the sheer audacity to then suggest that stuff like this is our problem because we don't do things the way they do.

Guess what: It *IS* your problem if your port crashes!

If I had to rethink everything I do for how it may affect things it's never meant for I wouldn't achieve anything.

If you don't want your port to crash - then do something about it - and don't blame other people's work for it.

Even for a plain vanilla WAD it's utterly careless to assume that all is well and will work. There can always be something wrong, either by bad tools or inept modders - and such badly constructed WADs will handle as badly as encountering a namespace you don't handle.

I still don't get though how this particular namespace poses such a big problem. The worst that can happen is that some oversized graphics are being used in place of smaller ones.

It doesn't do anything else - in particular it doesn't add and new graphics formats. Everything that can exist in HI_START/HI_END can exist in the global namespace as well.

And if this is a general complaint about how ZDoom dares to allow PNG in place of Doom's original graphics formats, sorry, then you don't have my sympathies.


--------------


A general note about the broken-ness of the WAD format:

If you people think that it's all well designed - just take a look at the crap some modders have made over the years and due to the insuffucient checks in the original engine still needs to be supported or old mods will break.

In my opinion any resource management that is so lax that it doesn't enforce its own rules is broken. And due to the backwards compatibiliy needs it's broken beyond repair if you ask me.

kb1 said:

#2. "Let's think about how we could possibly add this cool new feature, it a way where it can either be seamlessly added to existing structures, using existing code. If that's not possible, let's do some research: Probably there are some other ports in existance that also want to add features. Maybe we can collaborate, and devise a solution that can be considered a standard, so that eventually all ports can be improved. If we can, great - let's publish that standard. If we cannot, we will try again. We will repeat this cycle until there's no chance for agreement. When we have exhausted all efforts in collaboration, we will create the new standard ourselves, but we will make damn sure that our new standard would allow the other ports to easily use our standard. Because we are considerate and believe in compatibility, we will use the most simple structures and code possible."

And this is ZDoom's problem - how?

If you accuse one port developer you have to accuse all of them. I don't think there's not a single port in existence that's not suffering from the 'not invented here' syndrome in one way or another.

Also let's not forget that most of these things are old - very old in fact. They date back to a time when ZDoom was the only port even thinking about advanced features - a time when Boom was still the big player.

kb1 said:

And, one of my personal favorites, most Boom engines play original demos! Not only that, but they also play advanced Boom demos. Hell, PrBoom+ can play most all flavors of original Doom, Boom, MBF, etc. But, not ZDoom. Heh, ZDoom cannot even play most ZDoom demos...

So? Demos are not a major focus of ZDoom. Furthermore, the way they work they impose serious constraints on how you can change the engine.

I am seriously getting tired of people holding this against ZDoom. It's never been the focus of the engine and never will be. Just ask Quasar how many old cruft he needs to keep around just for demo compatibility. It's not something that comes for free - it's something that needs to be actively maintained and each and every change to the play engine needs to be carefully considered to avoid breaking old demos.

If you want to play demos, use PrBoom+.

[DaniJ]

Graf Zahl said:

Guess what: It *IS* your problem if your port crashes!

I will reiterate because you clearly still don't get it. The only reason my port crashes is because ZDoom later introduced a feature that is inherently incompatible with the logic employed by the original game. Your HI_START/HI_END construct dupes incompatible ports into loading data intended for your port when it expects something else.

Now, speaking of the ability to put PNGs in the flats "namespace" (which isn't the issue I might add) - how can you on the one hand agree with me that DOOM's Flat format is semantically not an image format (see earlier in this very thread) yet then say it is right and proper to include along side it image formats like PNG? So are you now saying that you think Flat is an image format? If so you can't really argue against the format Quasar proposed as the original topic of the thread.

If I had to rethink everything I do for how it may affect things it's never meant for I wouldn't achieve anything.

Generalised, obtuse over simplification - I won't comment.

If you don't want your port to crash... {snip}

More efforts to pass the buck. Yeah, I will do something about it. However, why should any of us sit quietly and not complain about the work you cause us?

[Graf Zahl]

DaniJ said:

I will reiterate because you clearly still don't get it. The only reason my port crashes is because ZDoom later introduced a feature that is inherently incompatible with the logic employed by the original game. Your HI_START/HI_END construct dupes incompatible ports into loading data intended for your port when it expects something else.

And again: The only reason your port crashes is because it can't handle a malformed WAD and chokes on it. That the supposed malformation is an unsupported feature of another port doesn't really matter. The same thing could as easily happen if a modder added some broken and misnamed stuff by accident. Your port is not supposed to crash as a consequence but instead I'd expect a meaningful error message.

Furthermore, HI_START/END is *NOT*(!!!!) an optional graphics replacement. Anything in there is supposed to be loaded no matter what. And it's most certainly not meant to add hires stuff to a mod that's supposed to work with other ports than ZDoom.

For that the recommended approach is (and always has been) to make this stuff optional.

And even if that is a problem - all that happens is that you get the wrong graphic lump - which brings us back to the PNG issue (a.k.a. missing validation checks)

DaniJ said:

Now, speaking of the ability to put PNGs in the flats "namespace" (which isn't the issue I might add) - how can you on the one hand agree with me that DOOM's Flat format is semantically not an image format (see earlier in this very thread) yet then say it is right and proper to include along side it image formats like PNG? So are you now saying that you think Flat is an image format? If so you can't really argue against the format Quasar proposed as the original topic of the thread.

Where's the problem?
PNG and JPG, and to an extent even PCX, TGA and Doom patch can all be validated by checking the lump - so if the lump gets validated as a recognized graphics format it's used as such - if not it's treated as a raw flat. Any perceived insanity in such an approach is owed to the cursed thing known as 'backwards compatibility'. I'd like to do it differently but since Doom doesn't care about the lump size the available options are very limited.

[DaniJ]

I don't agree with the conclusion you are drawing. How is it my fault that another application has taken an established format, built upon it by shoe-horning in additional implied functionality and causing a backward compatibility issue in the process? Drawing on another example from this very thread, you agreed with me about the possibility of Frankenstein PNG images - how is this so different?

Ok, I get that you say HI_START/END is not intended as being optional. However this is a semantic change from the basic concept of the feature as first devised. One which was implemented in other ports long before ZDoom picked it up. Is this my fault also?

Re: PNG vs Flat validation

Its not a problem for me personally as we've already accepted the fact that we have to (and indeed do) adopt the same approach used by ZDoom. However, this further underlines the point(s) being made - that cross compatibility is not even on the radar, let alone a consideration.

[Graf Zahl]

No it sure is not on the radar if that means to leave out features just to make you happy.

Mods for ZDoom are not meant to be played with Doomsday.
Whether that means gameplay errors, an error message upon start, an error message upon map load or even a crash is really not ZDoom's concern. If we had to consider each other port's reaction to newly added resource related features, we wouldn't be able to implement those without jumping through hoops. And the WAD format simply doesn't have any such hoops available without obfuscating the WAD directory.

Still, if your engine crashes if it finds data it cannot deal with it's certainly your problem, not mine. The least I'd expect from a well designed application is to output a message telling the user what is wrong. Crashing on unexpected data is always - in each and every case - a design flaw. In case of Doom I'll grant that some things are impossible to validate - but that's clearly not the case here, since we are talking about graphics. Why does it crash anyway? The 2 things I can imagine:

- incompatible graphics format. Solution: add checks to validate the graphics
- duplicate lump names causing the wrong lump to be loaded: Improve your lump management. This is something I can forgive vanilla Doom because it was never really designed for user mods - but in an open source engine made to play user made content falling in such a trap is absolutely unforgivable. Not because of ZDoom's addition of a new namespace but because of the fact that this can also happen by simple user error.

[DaniJ]

I'm not saying this is your problem. The whole point in my highlighting this was to try and get you to think about this kind of thing in future.

Validating the graphic format is not an issue (for our project at least).

Yes its a design flaw that this construct can cause a crash. Yeah a user most certainly could orchestrate something similar through simple error but that is rather beside the point.

This is a feature which achieves this through its very design. That is a completely different situation and one which I believe should have been considered in the design process (if you actually do care about cross compatibility). Particularly given the type and usage of the data involved.

However I get the impression that none of this is being absorbed and ZDoom will continue on its merry way, breaking standards and then telling everyone else to "fix" their code after the fact. How you can defend that approach I really don't know.

[Graf Zahl]

Ok, if I condidered your point and checked each new feature for cross-port compatibility...


- I couldn't use Hexen format maps
- I couldn't use WAV, MP3 or Ogg as sound lumps
- I couldn't use MOD, XM, MP3 or Ogg as music lumps
- I couldn't use PNG or JPG as graphics lumps

Each of these will crash a more 'traditional' engine.

I hope you see the problem of your request.

And what for? That some idiots can load the mod with an engine it's not designed for and complain that it doesn't work?

I think we all have to accept that mods being made for specific engines won't work with others. I certainly wouldn't want limit my options just to avoid those mods crashing those other engines.

Sorry, but this is a consideration I wouldn't be able to sell to my customers (i.e. ZDoom's users) because it would limit their options.

I'd prefer if everybody used Zip/PK3 instead of WAD but that's clearly not doable.

[Gez]

DaniJ said:

I will reiterate because you clearly still don't get it. The only reason my port crashes is because ZDoom later introduced a feature that is inherently incompatible with the logic employed by the original game. Your HI_START/HI_END construct dupes incompatible ports into loading data intended for your port when it expects something else.

From what I understand of the Doom engine, for this to happen, the mod has to be constructed somewhat like this:

F_START
...
HI_START
...
HI_END
...
F_END

Or, even worse, like this:

HI_START
...
F_START
...
HI_END
...
F_END

Then it's more a question of the modder being illiterate about it. If the mod is constructed properly, without overlapping or embedded namespaces, there is no reason that a search for the flat WHATEVER should begin anywhere except at F_END - 1 and end at F_START + 1, so it there is a WHATEVER lump in the HI_ namespace it shouldn't be found.

[DaniJ]

Doomsday was doing all of those things (MD2, PNG, OGG, etc...) using Zips many years before ZDoom decided it would be a clever thing to do to try and cram them into WAD namespaces for semantically different data. Skyjake even went to the trouble of devising a simple mechanism for that data to be put inside a WAD if the author really wanted to do so, which doesn't break other ports in the process.

Along comes Zdoom several years later and in what seems like a massive fit of NIH - implements all the exact same stuff in a completely incompatible manner from the ground up. And considering the WAD format is so "completely broken" as you put it, I'm left wondering why??

It wouldn't be so bad if it were a one off. However every port author actively developing a DOOM port can attest as to how frequently this sort of thing happens.

Thus, I can only draw one conclusion - you care nowt for cross port compatibility and any time the rest of us spend on extending your project such curtsey is completely pointless.

@Gez
I don't recall the details of the HI_START/END problem off hand so I can't say one way or the other. However this issue itself is not what I particularly care about, its just one in a long line.

[Graf Zahl]

DaniJ said:

Doomsday was doing all of those things (MD2, PNG, OGG, etc...) using Zips many years before ZDoom decided it would be a clever thing to do to try and cram them into WAD namespaces for semantically different data. Skyjake even went to the trouble of devising a simple mechanism for that data to be put inside a WAD if the author really wanted to do so, which doesn't break other ports in the process.

What 'simple mechanism'? Since breaking other ports is the issue here it has to involve mangling of the lump names - or the same 'problems' would have surfaced.

What I don't get is why you even care. The feature sets of ZDoom and Doomsday are so radically different that the mod wouldn't run anyway.


As for the NIH with Zips, I had a look at the way Doomsday did it - and what I understood simply did not meet ZDoom's requirements - not to mention that at the point in time I was unable to find any usable documentation. I still have trouble finding some comprehensive docs about how Doomsday Zips need to be constructed.

So with the bare information I had I saw no point pursuing it. And so far nobody has lost due to it. The only common denominator between ZDoom and Doomsday is limitless vanilla - and anyone mapping in that will use neither ZDoom nor Doomsday features, and in particular would create WADs that are designed for all ports.

So, ultimately I really don't get where your problem is. Your engine doesn't support ZDoom features so there's no need to be able to load ZDoom mods. If during load it throws an error telling the user that it contains unexpected data, it's more than enough. The user doesn't need to get any further with it.

Oh, and just in case you ask: What do I expect when loading mods made for other ports into ZDoom?

- best case scenario: the mod loads but the maps are not playable. - worst case scenario: a crash due to some unexpected data being found. If that happened I know that some code is missing a validity check and I'd add that check ASAP to replace the crash with a meaningful error message. What I wouldn't do is to complain about some other port daring to allow data in the WAD that wasn't supposed to be there.


I sure am not that naive to assume that since I got a WAD file it can be loaded properly.
WAD is merely a container format that can contain anything, be it graphics lumps, music, sound effects and so on - most of it mashed together in one unsorted namespace. So how to deal with it? The only answer can be to assume nothing and check everything

[Blzut3]

DaniJ said:

However I get the impression that none of this is being absorbed and ZDoom will continue on its merry way, breaking standards and then telling everyone else to "fix" their code after the fact. How you can defend that approach I really don't know.

OK lets put the formats issue aside since while I do see your point, I do not think it's proper to make the user think about what format their resource is in and then have a separate mechanism for loading that kind of resource. Furthermore from a look on the wiki, Doomsday's solution was to map wad lumps to full filenames. This would not have worked for us since ZDoom had support for other file formats before the pk3 format was around IIRC. (IIRC even vanilla supports having midi and mus data for music which I believe caused problems with Doom95 expecting mus only? Basically the same thing here.)

The other issue mentioned is the addition of new features to Hexen lumps. As I already stated, I do think there could have been better cooperation on this, at the time this started the number of Hexen ports were small so posting about it here would have resulted in a bunch of "I don't really care since my port doesn't have Hexen support." I realize Doomsday likely got Hexen support around the same time, but the point remains that it was less of an issue back in the day. Looking forwards I would like to increase compatibility between ports as it is only recently that we're starting to see mods which target multiple ports and the needless duplication of features makes life harder for mod authors. That said, I believe it would be trivial for a port to recognize unknown syntax in SNDSEQ, ANIMDEFS, and SNDINFO and error out if vanilla doesn't do so. (Unfortunately Raven didn't give any of the lumps adequate syntax to ignore unknowns.) MAPINFO could similarly be programmed to error out, but we've already tried to solve the problem with introducing ZMAPINFO (which can ignore unknowns).

Looking at the pk3 stuff in Doomsday, I don't see a significant difference between the supported directories other than sounds/ in ZDoom seems to be named sfx/ in Doomsday. That said I can't really draw any conclusions since the wiki appears to be missing any useful documentation of the Doomsday's file system.

[Gez]

Blzut3 said:

I realize Doomsday likely got Hexen support around the same time

Historically, Doomsday got Hexen support before it got the Doomsday name. It started as jHexen. Then jHeretic and jDoom were derived from it, and later Doomsday appeared as the unification of the three.

[DaniJ]

Graf Zahl said:

What 'simple mechanism'? Since breaking other ports is the issue here it has to involve mangling of the lump names - or the same 'problems' would have surfaced.

Correct it does involve a redirection mechanism where a lump name is mapped to a virtual file system path using a special lump (DD_DIREC) which contains the mappings.

What I don't get is why you even care. The feature sets of ZDoom and Doomsday are so radically different that the mod wouldn't run anyway.

I care because users do not consider the technical details when they attempt to load mods. Users do not think "ah this mod is designed for ZDoom so it won't work in Doomsday" they will just attempt to load it anyway and then report the issue as a bug when it doesn't work.

As for the NIH with Zips, I had a look at the way Doomsday did it - and what I understood simply did not meet ZDoom's requirements

I find this very hard to believe considering Doomsday places no restrictions on the layout of Zips what so ever. In our implementation the mod author is free to layout the resources however they wish.

...not to mention that at the point in time I was unable to find any usable documentation. I still have trouble finding some comprehensive docs about how Doomsday Zips need to be constructed.

See previous point (no restrictions, only a set of automatic mappings).

I sure am not that naive to assume that since I got a WAD file it can be loaded properly.

You might not assume that but a great many users do.

[wesleyjohnson]

I have to agree with DaniJ, Gez, and others about zdoom-isms. They are like virus-wads in that they hide incompatible data. I am not fond of having to research each problem wad to discover if it is has a zdoom-ism.

I despise the argument that the ONLY alternative to what zdoom did was to not do anything. That deserves to be ridiculed. A unique marking of the format was all that was requested and that would not have hindered it at all. Something like changing one letter of the markers to be unique would have cost another 1/2 hour of development time to duplicate some detection lines.

Every extension to the wad format needs some kind of identification and differentiation. ZDoom should have marked their incompatible wads as PWADZDOOM at the top. That file identification "PWAD" is not just decoration. I do not accept the evasions, the doom community has been harmed by their need to deal with the fallout, however it expresses itself.

DoomLegacy once moved the 3D-floor extensions to get away from linedef codes that were being used by some other port. I was not aware that we are still overlapping with some other ports usage.
My detection of Boom vrs DoomLegacy consists of flagging when any such special is encountered during loading. Detecting Boom, but no DoomLegacy extensions enables some Boom specific emulation.

I am working on the FOG linedef extension (special=302).
For additional parameters it uses texture names that look like "#128G", which gets decoded into alpha=128, fogtype='G'.
Because such names are already in use for other linedefs, I assume that such texture names are not a problem for other ports.
For the purposes of this argument, please note how they are marked with a "#" for detection, and only appear as texture names for the special linedefs. They appear to be a unknown texture otherwise.

I try to make DoomLegacy play ANY wad, if there is a way. Throwing an Error for any problem is no longer acceptable. I will substitute texture[1] for a PNG if I have to. It may look funny but the wad will play. The user can then see if it is worth additional effort.

My concern with PNG was to get ahead of the incompatibilities by having this forum state the proper uses and unacceptable uses before the wad authors create weed wads (unwanted aberrations).

[Gez]

wesleyjohnson said:

Every extension to the wad format needs some kind of identification and differentiation.

The WAD format is absolutely unchanged.

wesleyjohnson said:

ZDoom should have marked their incompatible wads as PWADZDOOM at the top.

This is silly. ZDoom is not a WAD editor.

Does Legacy require its maps to be created in a different, incompatible WAD format (header doesn't have the same size since it's PWADLEGACY instead of just PWAD), meaning that no existing map or resource editor allow to create them? Does Doomsday require PWADDOOMSDAY?

There is no difference between saying "ZDoom should require WADs to be PWADZDOOM" and saying "nobody should make mods for ZDoom". Let's be serious one moment here.

[Graf Zahl]

'Silly' ist a major understatement.

WAD is merely a container format, nothing more, nothing less. There's nothing said about its contents. Theoretically it can be anything. Hell, WAD was even used by other games for completely incompatible content because the tools to manage the files existed.

Furthermore, where should this stop?
Should only ZDoom make an exeception with tagging or should other ports do as well? What should editing tools do about this?

Never forget that for your complaints other people not involved with your port would have to invest some work to implement such outlandish ideas. What for? To make your life easier with the attempt to 'make all files work'? Sorry, but that's ludicrous. Throwing an error when you discover that some unusable data is present is perfectly fine, in fact I believe it's an essential part to ensure robustness.

Just starting up at all costs, despite some problems being found on initialization is idiotic. You only hide potential problems from your users. From experience I can tell you that modders rarely check their creations for properness. Warnings get ignored and a semi-broken project gets released. Sometimes you have to be obnoxious and tell your user 'not like this - I refuse to run this!'

In ZDoom there's even some abort conditions explicitly coded in to prevent stupid modding practices from the outset.

Honestly, trying to make any WAD in existence for any port out there work in yours is a futile endeavour. There's so many variations, aberrations and never thought of ways to do things wrong that no matter how hard you try, eventually you'll fail

[Phobus]

From a player/modder perspective I am, frankly, amazed that people try to run ZDoom mods in Doomsday (or vice-versa). Surely the reason we include text files in our downloads is so that people can identify what source port to run the mod with? You shouldn't defend that action, you should just tell people submitting these "bugs" that they're trying to run a mod in the wrong source port and can't expect it to work. Best case scenario for users would be for the port to be able to come up with an error message recommending the correct port for the mod, but that honestly sounds impossible to do accurately.