Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Quasar

Simple linear format

Recommended Posts

The biggest problem I see with PNG is that palette. Having its own palette means that it could be different than the Doom palette.
Using generic tools on the PNG file would make it likely.

If there is a rule that it must only be the Doom palette, then the palette is not needed in the file.

Could post process all the PNG to remove any palette and fix any PNG with the wrong palette to use the Doom palette. That would make for smaller overhead and remove the palette issue when loading the PNG.

This would have the advantage of not requiring any editor changes.
Just need a PNG-Doom-palette-compressor program to process the PNG before it gets incorporated into the WAD.
For the editors that build their own wads, could have a version that does the PNG palette stripping on existing wads.

Would the PNG library tolerate loading such a tweaked PNG ??
Could the PNG palette be optional, or faked-out with a one entry PNG palette ??

Similar problem, XML and PNG are generic exchange formats, designed to handle everything. When used for large amounts of data, with heavy restrictions on format and content, the generality and its overhead is no longer an asset but becomes a problem.

Share this post


Link to post

Ugh...

Why cripple your images? If the engine can do it it may be better to leave the original colors intact for true color capable engines, like hardware renderers.

Share this post


Link to post

Agreed. As I mentioned earlier, one important thing about PNGs (the most important one as far as I'm concerned) is that they can have their own palette. This allows a modder to use a PNG in their WAD and have it look perfect in a hardware/true colour port but (assuming the port being used can do this) it will be mapped by the engine to the nearest matches in the game palette.

If PNGs didn't allow this, I wouldn't see much reason to use them at all and I'd just go with the original Doom format because it's properly supported by everything. I certainly wouldn't bother with a new format that gave me no particular benefits.

Share this post


Link to post

That'd all be fine if it were possible to reliably handle requantized images with translation tables, but it's not.

EE will not be following ZDoom in replacing translations with auto-calculated hue shifting because it washes out the text (note how the red "v" font and big font have no shades of brown in them in ZDoom). Hand-tuned translations make for much prettier and more readable text.

Share this post


Link to post
Quasar said:

Hand-tuned translations make for much prettier and more readable text.

Perhaps. On the down side we have loss of true color support (in renderers that support it) as well as increased burden on the mod author should they decide to change the palette. In addition you place a restriction on what format fonts can be in (not that you couldn't have the best of both world though).

Share this post


Link to post

Don't forget that ZDoom uses true color 3D accelerated rendering for all HUD graphics. As a result the recolored fonts are no longer tied to the game palette and therefore not affected by lack of matching colors.

And you cannot easily create your own font colors with a simple definition lump if you keep the palette dependencies so close to the surface.

BTW, even ZDoom clearly says that in order to use translations on sprites you got to use Doom patch format to guarantee good results - but nobody ever drew the conclusion from that that a new graphics format was needed.

Share this post


Link to post

On another game project I had reached a similar conclusion, that it would be RGB graphics based, and any palette-based renderer would have to cope. DoomLegacy has started to adapt too, giving at least equal importance to RGB in that the new Fog effect may use Draw32 as the reference implementation.

But with so many Doom ports critical of differences from vanilla,
adopting PNG as a format that supplies RGB (non-Doom-palette) textures will have wildly differing appeal. It certainly needs to be stated that the palette of such PNG can be other than the Doom palette. The re-quantization of such images for palette based drawing will be necessary, and everybody needs to know that.

Still have the question of whether a PNG can be without palette, and would the PNG library tolerate such ??

It certainly sounds like those who actively want non-Doom-palette textures are not going to be supporters of a standard header that does not support that.

Share this post


Link to post

The issue I have with adding a header to DOOM's Patch format for the purposes of graphic format recognition, while well meaning, is (IMO) somewhat misguided. We aren't talking about a graphic file format that just so happens not to have identity information. As has already been mentioned in this thread, its purpose is as a "ready-to-render" format optimized for fast rendering by DOOM's software renderer. Semantically the Patch format is roughly equivalent to a format used by hardware graphics drivers, either deep in the driver or with textures actually resident on the graphics card in video memory.

Consequently I believe that adding general purpose image-format like information to it (magic bytes, palette info etc...) rather goes against the original purpose of the format.

In the same vein, going in the opposite direction, by creating a Frankenstein indexed PNG sans palette is just as illogical.

Share this post


Link to post
DaniJ said:

In the same vein, going in the opposite direction, by creating a Frankenstein indexed PNG sans palette is just as illogical.



Indeed.

The whole thing smells too much like a solution in search of a problem.

It's so simple: If you want palette-less patches - use the Doom patch format.

If you want flexibility, use PNG.

Patches not having a header doesn't really count. With some sanity checks on the patch data it's relatively easy to discover broken ones and handle them appropriately.



The only reason to support other formats is if those offer some true advantage - JPG is a good example for space saving reasons sometimes. All other formats supported by ZDoom are there because they were needed for some stuff (I implemented PCX and TGA only so that some older models could be loaded with their skins - the formats themselves offer nothing of interest otherwise.)

Share this post


Link to post

I don't know why people are now talking about adding a header to the patch format. This seems especially pointless. Patches already have a header (read in the patch_t structure).

This isn't what Quasar was talking about originally. More like adding a header to the flat format, so that they don't have to be 64x64 or some other magic number.

Share this post


Link to post

In this context there is no difference between the Flat and Patch formats. Exchange the word "Patch" with "Flat" in my previous post and all the same arguments still apply.

However, I did use Patch where I meant Flat.

Share this post


Link to post
Gez said:

I don't know why people are now talking about adding a header to the patch format. This seems especially pointless. Patches already have a header (read in the patch_t structure).

This isn't what Quasar was talking about originally. More like adding a header to the flat format, so that they don't have to be 64x64 or some other magic number.



So what?
This whole discussion seems to imply that the data has to be stored in a ready-to-use format because decompression/conversion is prohibitively expensive. This is utter nonsense. Decompressing a PNG - in particular those with sizes relevant for Doom - takes considerably less time than loading it from the resource file - which renders the entire discussion moot.

And if all else fails - add some checks to allow patches in the flat namespace and all is well again.

Share this post


Link to post
Graf Zahl said:

So what?
This whole discussion seems to imply that the data has to be stored in a ready-to-use format because decompression/conversion is prohibitively expensive. This is utter nonsense. Decompressing a PNG - in particular those with sizes relevant for Doom - takes considerably less time than loading it from the resource file - which renders the entire discussion moot.

And if all else fails - add some checks to allow patches in the flat namespace and all is well again.

I never said it was prohibitively expensive. Only that it's *more* expensive than loading a linear resource. That's not really deniable. It's whether it matters or not that is up for debate, and likely it does not until the point it was multiplied by thousands of resources. Then EDF loading might take a few seconds instead of a blip. Boo hoo I guess.

I'm already over the idea. There's no use continuing to call my ideas ridiculous or nonsensical as if you are going to dissuade me. I already committed code to read PNGs as linear font lumps like, three weeks ago. :P

EDIT: In case you really need the gory details to be satisfied:
http://mancubus.net/svn/eternity/trunk/source/e_fonts.cpp?op=diff&rev=2202&peg=2202

Share this post


Link to post

I am with you Quasar in wanting the play time free of texture reload blips. I did not find the discussion to be worthless.

Each port will make its own tradeoffs, but I do not expect mine to be played on 3Ghz machines with 2G of main memory.
If a texture is not in a readily usable format at load time, I would want to convert it to some readily usable format. Some of the texture conversions are not as cheap as just decoding the PNG compression. Palette closest color search would be one case, which is forced by PNG palettes.

I currently just mark texture conversions as higher-priority memory, which is not unloaded unless first pass cache flushing fails to deliver memory.

Share this post


Link to post
Quasar said:

..I'm already over the idea. There's no use continuing to call my ideas ridiculous or nonsensical as if you are going to dissuade me...

Of course, you did ask for it - by asking. :) Which, is unfortunate, because discussions of this nature are important, and I, for one, do appreciate them.

As far as the PNG thing goes, here's my 2 cents: My biggest problem with the whole ZDoom/.PNG/.PK3, etc. thing, among others, is that, I feel that ZDoom went and added all manners of specialized formats for this and that, without asking the community, and without the first thought about compatibility across ports.

I've been working on a port for quite a while, carefully adding support for some of the common features of the modern ports, but, I have a hard time justifying adding support for many of these specialized features, just for the sake of ZDoom compatibility. I just don't see why many of those drastic deviations from the original were necessary.

Not only do ports have to add support for PNG, flats on walls, PK3, compressed nodes, 15 music formats, etc., but, wad tools also have to be able to handle these things. Sure, some of these are nice features, but at what cost? For example, nowadays many people don't use the tag entries, like S_START anymore *because it doesn't matter anymore*, but that means that wad tools can't tell what is what quite as easily.

Sure, I could "just play ZDoom", but my port has some features that ZDoom lacks. (Of course, when I eventually release it, ZDoom will have support for my features in a week, and then it will expand upon them in an incompatible manner...)

Such changes could have been done in an open, structured, and documented manner, with a prejudice against adding more than necessary to get the job done.

I could elaborate on this point for hours, but, I'm done. </rant>

Anyway, I just wanted to state that I appreciate such discussions, even if they don't always pan out.

Share this post


Link to post
kb1 said:

As far as the PNG thing goes, here's my 2 cents: My biggest problem with the whole ZDoom/.PNG/.PK3, etc. thing, among others, is that, I feel that ZDoom went and added all manners of specialized formats for this and that, without asking the community, and without the first thought about compatibility across ports.

Of course ZDoom is usually the first port to adopt any such standards. :P

That said a few points to make. First a lot of the features you mention (support for many sound/graphics format/mixing, etc) are a direct result of organizing the code such that all of that can be gotten "for free." The sound formats in particular are mostly because fmod supports them rather than code in ZDoom itself (this does not apply to synthesized formats, but for the most part you can be ZDoom compatible without supporting any of the esoteric ones). Not sure what you mean by people not name spacing their entries since it does indeed matter in ZDoom.

The only thing you could argue a case for is the many scripting languages in ZDoom. For the most part it is the way it is because we would never agree on a universal scripting language for one reason or another. You could argue that we should have been more open on the design for TEXTURES, and I would probably agree with you, but that's the exception.

Of course there's also the problem that ZDoom has a lot of features that wouldn't apply to most ports. At least at the time the features were developed. Case in point, how many ports support Hexen? Since few do so is it worth openly discussing every extension to the Hexen script formats?

Share this post


Link to post
kb1 said:

Such changes could have been done in an open, structured, and documented manner, with a prejudice against adding more than necessary to get the job done.



This could never, ever work in the developer community because you have 10 programmer egos to deal with. The only time I can remember where it ever worked out was UDMF and look where we got: Aside from ZDoom and its children there's still no other port supporting it.



wesleyjohnson said:

I am with you Quasar in wanting the play time free of texture reload blips. I did not find the discussion to be worthless.

Each port will make its own tradeoffs, but I do not expect mine to be played on 3Ghz machines with 2G of main memory.
If a texture is not in a readily usable format at load time, I would want to convert it to some readily usable format. Some of the texture conversions are not as cheap as just decoding the PNG compression. Palette closest color search would be one case, which is forced by PNG palettes.


That's the completely wrong approach to do things. If you know there's a bottleneck you investigate ways to optimize it, not to avoid it first. In this case it isn't even that hard: Create the mapping tables when you precache your texture for the first time or when you preinitialize it.

In ZDoom these remapping operations never really affect in-game performance because they are all done up front when the textures are initialized - and kept in memory.

Sure, if you got lots and lots of textures it may cost a MB of RAM or so but that really only gets a problem if you are dead set on supporting 10+ year old systems.

And as I said before, all the decoding time is easily made up by shorter HD loading times. I did some extensive profiling on this when I added PNG support to GZDoom's OpenGL renderer and all the time spent initializing the texture data was mostly irrelevant compared to the time it took to load it off the resource files - and this was years ago on a system considerably weaker than what we got today.

If you want to avoid loading delay there's basically only one way to do it: Cache your data! Cache as much as you can as early as you can.

In realistic terms, even the weakest system in use today has enough memory so this shouldn't be an issue. The memory is not there to be left alone. It's there so that applications can use it to improve performance. Trying to keep the memory footprint at a handful of megabytes at all costs is utterly counterproductive.

See it this way: I still got an 8 year old computer lying around with a 3 GHz P4 and 512 MB of RAM. That wasn't even top of the line when I bought it but it's still more than enough to run Doom with software rendering at acceptable speeds in high resolutions with memory intensive mods.

I didn't retire this computer for performance, not even game performance, but because over the years even 'normal' applications stopped working fine with only half a GB of RAM. So if you got a computer weaker than this, I'd consider it mostly unusable now. Yes, it still runs Doom or other ancient games - but that's pretty much all it runs. Not even a web browser works fine anymore thanks to all the nonsense that's done in the WWW these days.

And now seriously: Why should I set my minimum requirements to something even lower than a system that barely does its job today?
I'd only punish those who got decent machines in favor of the few that still run outdated hardware. I want to give the 90% that have a good system the best possible experience, not cater to the 10% who fall through the cracks due to ancient hardware.

In similar terms: ZDoom has recently removed Windows 95 support.
Why?
Simple answer: Nobody needs it anymore!
It had been broken for quite some time with absoutely nobody reporting it. It was just discovered by accident that it didn't work anymore.

So, some of you guys may cry foul that Randy dared removing it altogether but let's be realistic.
Is it worth investing time getting it running again?
That depends on only one factor: How many people are using it? The answer from the download statistics said it was an irrelevant fraction of one percent? Not worth the time. The same thing applies here. At some point you have to draw the line. When does the sacrifice for supporting old hardware become an issue.

Even commercial games more than 10 years ago stopped giving up having their data in preconverted formats on disc and opted for using common graphics formats instead. This was clearly because the developers realized that the added development burden of having to deal with custom made tools to handle these formats didn't pay off.

This all doesn't mean that you shouldn't optimize - but if you optimize do it in places where it really counts and really pays off, not in the places where it may seriously affect your customer's ease of use of your product.

This kind of attitude should be a thing of the past.

Share this post


Link to post
Graf Zahl said:

See it this way: I still got an 8 year old computer lying around with a 3 GHz P4 and 512 MB of RAM. That wasn't even top of the line when I bought it but it's still more than enough to run Doom with software rendering at acceptable speeds in high resolutions with memory intensive mods.

I didn't retire this computer for performance, not even game performance, but because over the years even 'normal' applications stopped working fine with only half a GB of RAM. So if you got a computer weaker than this, I'd consider it mostly unusable now. Yes, it still runs Doom or other ancient games - but that's pretty much all it runs. Not even a web browser works fine anymore thanks to all the nonsense that's done in the WWW these days.

Sounds like my machine, though mine is still in better shape than yours. :P

Share this post


Link to post
kb1 said:

For example, nowadays many people don't use the tag entries, like S_START anymore *because it doesn't matter anymore*, but that means that wad tools can't tell what is what quite as easily.


I'd blame NWT for that, not ZDoom.

Share this post


Link to post

Yes, unfortunately the archives are full of WADs that were lazily or incorrectly put together but they worked. As a result, ports have to be lax to support these WADs. If Doom had been strict about what it accepted then the current situation wouldn't exist. However, extensive modding of Doom was not a priority when Doom was being made and no one realised the extent that modding would get to so it is perfectly understandable.

Share this post


Link to post

Let me reiterate that UDMF is on the slate for EE. It's entirely my fault that the linedef special system is so screwed up in EE, but I caused it to become that way many many years ago when it wasn't obvious that any of this would happen, and I knew a lot less than I do now :P

Aeon is going to be done first, it is in progress right now. It is being done first to relieve the native inventory system of needing to support such bogus crap as items with multiple effects that have to be iterated over before, during, and after collection. If a custom item needs to do more than what one of the parameterized native item effects can account for, it'll just be scripted.

EE's quip will soon be "Do it in Aeon, " vs. ZDoom's "Wait for DoomScript".

After inventory, weapons and keydefs will be carried out immediately. This completes everything blocking the implementation of the rest of the idTech 1 commercial games, and means that it's time to redesign the linedef and sector special systems to account for them. That's when UDMF can be implemented. It is a roadmap that I have been following for the past year and half, with some adjustments and reprioritizations as they've become necessary (see the consecutive failures of edf3-branch and inventory-branch to achieve functional inventory...).

Share this post


Link to post

- This rambles and never got honed to a point, as I have to go help someone -
One of my concerns is how to deal with wads that have PNG and other strangeness (another wad variation to deal with).

I have in DoomLegacy some detection of special linedefs to differentiate Boom wads from DoomLegacy wads. For instance, the way that colormaps are handled changes if the wad has only Boom special linedefs.
I see more of that being needed for PNG textures. Is it a ZDoom-only wad ?, or did the author just throw in a PNG because they heard that they were neat ? The depth of support a port gives to PNG can vary wildly too, and vary with which render mode is being used (for those who have them).

Cannot expect the wad authors to constrain themselves to predetermined good combinations of features. There has not been a limited resource in the program that has not been exceeded by some determined wad author (segs, vis, planes, etc). What are they going to do with PNG ??

Share this post


Link to post
wesleyjohnson said:

I see more of that being needed for PNG textures. Is it a ZDoom-only wad ?, or did the author just throw in a PNG because they heard that they were neat ? The depth of support a port gives to PNG can vary wildly too, and vary with which render mode is being used (for those who have them).



This should be very simple: Don't try to second-guess. If you can handle a texture, load it. If you can't, throw an error. Not necessarily a fatal one but at least a console message 'Texture xxx could not be loaded.' That's all ZDoom has in place here.

Share this post


Link to post

While conceptually I agree with you Graf, the problem wesleyjohnson is likely alluding to is that since the introduction of new texture namespaces such as HIRESTEX by other ports - his is now crashing when presented with a wad which uses them.

Essentially a port author is required to make the same leap in the management of textures and that isn't exactly trivial (particularly for those written primarily in C).

This goes back to the points raised earlier about inconsiderate design.

Share this post


Link to post

So what, ports should restrict their feature set if implementing a new feature means that other ports would no longer be able to load such mods?

Now that doesn't make sense

Share this post


Link to post

Yes. The responsible thing to do is to ensure your enhancements to a widely used format/standard do not break other apps. You'd be the first to complain if the shoe was on the other foot and don't pretend otherwise.

Share this post


Link to post

In other words: the first thing ZDoom ever did (implementing support for Hexen maps) was wrong.

Even Boom made significant changes to the way WADs are handled and WADs made with Boom in mind won't work on non-compliant engines.

The original WAD format was so broken that sticking to it would have meant not being able to advance the engine much.

Sorry, but you don't make any sense at all. With that attitude no progress would be possible.

BTW, what should I care if a Doomsday mod won't load properly in ZDoom? It doesn't work anyway.

It's naive to assume that any port can gracefully handle specific features of other ports.

I make sure that I can load what I want to support and wouldn't bother with the rest. So if some port developer chose to support BMPs for textures, those textures would produce error messages in ZDoom.

And if your port crashes on unknown data formats my only remark would be to implement better validation routines. It's the dumbest thing to assume that any lump that's of type x will be correct. Not only may some ports add new format support, it may also choke on broken data - which were generated by some ancient tools.

(BTW, the only texture format that won't get validated by ZDoom is raw flats in the flats namespace - because the way they get used makes it impossible to validate them - but ZDoom doesn't accept any other graphic that fails to validate as a supported texture format.)

Share this post


Link to post

Supporting Hexen maps is not relevant because DOOM ports won't be mislead into thinking they are DOOM maps if they support both.

BOOM did indeed make significant changes however those changes AFAIK did not result in another port crashing should the user happen to load a mod which used them. They failed gracefully even when loaded in vanilla.

Your comments about the WAD format being broken and not-extendible are absolute rubbish. Case in point - Doomsday was doing high res texture replacements many years before ZDoom decided to implement support and Doomsday also features ways to include them in WADs which do not break other ports.

With the HIRESTEX namespace it isn't simply a case of checking for whether a texture is a PNG or not because it is mislead into loading a PNG (or other) when it expects to find a Flat. Due to the utterly careless way in which this was implemented, any port which does not support both BOOM's namespacing mechanics and has explicit support for this namespace will immediately crash when the user tries to load a mod which uses it.

The whole point behind high res texture replacements is that they are optional replacements.

Share this post


Link to post
DaniJ said:

With the HIRESTEX namespace it isn't simply a case of checking for whether a texture is a PNG or not because it is mislead into loading a PNG (or other) when it expects to find a Flat. Due to the utterly careless way in which this was implemented, any port which does not support both BOOM's namespacing mechanics and has explicit support for this namespace will immediately crash when the user tries to load a mod which uses it.

The whole point behind high res texture replacements is that they are optional replacements.



I think you fundamentally misunderstand this namespace. It is in no way meant to contain anything optional. Replacements from this section will be used unconditionally.

And if your port crashes on unexpected data, it's time to rethink your implementation. The worst that should happen is an error message telling the user that the mod can't be used - which is the most I'd expect anyway in this situation.

As one film once aptly put it, Assumption is the mother of all f*ckups.
So if a port crashes due to being fed unusable data I don't blame the data, I blame the insufficient validation code. (In your particular case it'd be assuming that a flat lump is large enough for actually being used as a flat.

BTW, any port not supporting Hexen map format encountering one such map will also crash if it assumes without validating that the map is in Doom format.


To sum it up:

I really don't care when Doomsday/Legacy/PrBoom or whatever else crash when loading a ZDoom mod. The mod wasn't made for these engines so behavior is inevitably unexpected. Trying to make ZDoom WADS compatible with the loaders of these engines would mean putting in lots and lots of restrictions and making the modders' job a lot harder (like, no PNG/JPG for graphics, no WAV/MP3/OGG for sound and so on.)

There's much worse crash conditions anyway aside from the unsupported namespace and if I had to consider consider any change's impact on other port loading a mod with this change we'd quickly get nowhere.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×