Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Blastfrog

PALREMAP lump idea

Recommended Posts

Several high profile WADs that use fancy palettes tend to be including wholesale converted IWAD assets. So I was thinking, Wouldn't it be better if these PWADs could have "IWAD translation tables" to do this step on startup instead of in the editor?

Not only would it reduce file size (and thus space and bandwidth), but it would also avoid the troublesome issue of it not technically being allowed to distribute unmodified assets. Sure, they're technically modified but let's not kid ourselves.

EDIT: New thread title, seeing as the conversation has taken a different turn.

 

EDIT 2: It's been a while. I'm changing the thread title back from "32-bit rendering discussion", as I want to refocus on the palette translator idea. I don't like it when I start threads and they get derailed entirely off-topic.

 

I don't like my idea of having it all limited to just the IWAD or whatever crap I said last year. Instead, it should just translate whatever was loaded previously, yes, even if you load multiple WADs with PALREMAP, translating accumulatively. In any WAD with a PALREMAP lump, it is assumed that the assets in that same WAD are already made for the palette inside of that WAD, it only translates what is in the WADs below it in priority.

Edited by Blastfrog

Share this post


Link to post

From a purely technical standpoint this should be no problem. The main issue is to convince all source port devs to implement such a feature.

There's also the added issue of true color ports, so ideally this feature should be doable in a way that those can use the actual IWAD palette rather than the translation lump.

So my suggestion would be a 257 byte lump where the first byte can be 0 or 1 where 0 means that true color ports can just use the original IWAD palette and 1 means that they have to completely remap all colors to the new palette, followed by 256 palette entries to remap the colors.
Ideally this can be followed by a custom palette, in case that the IWAD palette should not be used but simple remapping would not yield the best possible results if true color is available.

Of course the proposed format is subject to discussion, maybe it's better to put some identifiers in to make it more robust - but the most important step here is that all ports support it - it'd be useless if only one of the important ports would pass.

Share this post


Link to post

It probably goes without saying, but I generally have larger projects such as megawads or other involved mapsets in mind. I don't see there being much benefit in having more than one translation table at a time, since this is geared toward the traditional IWADs and not standalone TCs or anything like that.*

I hadn't considered true color, but that is certainly worth covering. Your proposed solution seems to cover everything quite adequately. Unless there are serious objections to it, I'd say it's worth sticking to what you described to avoid bikeshedding.

Perhaps you can be the first to implement it if you like the idea, others might be more likely to follow suit.

Other port authors, such as the people behind Eternity, Zandronum, Doom Retro, etc, what are your thoughts?

*EDIT: Hmm, maybe it'd be a good idea to apply it to anything in the autoload queue as well, in case the user is using something like the Minor Sprite Fixing Project. No other PWADs that were loaded manually, however.

Share this post


Link to post

Doubt it's that simple. I'm guessing the BTSX sprites have been carefully tweaked to look nice in the new palette?

Share this post


Link to post

I doubt that. From what I remember it was merely some minor alterations because the palette removed a few of the color duplicates and some of these were used in the sprites and had to be remapped.

Share this post


Link to post
fraggle said:

Doubt it's that simple. I'm guessing the BTSX sprites have been carefully tweaked to look nice in the new palette?

Maybe the "palette translation table" lump could be designed in such a way that it would support defining "special cases" to translate particular sprites differently than the "default" rest of IWAD sprites. Ideally a potentially unlimited number of these special cases.

Share this post


Link to post

Handling special cases doesn't seem like a good idea to me, it'd only make everything muddy and complicated and ensure that this hypothetical feature will never be adopted.

If a couple sprites can't make use of the provided translation, then just include them as they should be. If all sprites can't make use of the provided translation -- make a better translation.

Share this post


Link to post

I propose the lump should be named "IWADTRAN".

fraggle said:

Doubt it's that simple. I'm guessing the BTSX sprites have been carefully tweaked to look nice in the new palette?

It actually is that simple. Looking inside the WAD, they look like they're identical to the IWAD versions with the exception of palette indices being moved around.

Gez said:

Handling special cases doesn't seem like a good idea to me, it'd only make everything muddy and complicated and ensure that this hypothetical feature will never be adopted.

If a couple sprites can't make use of the provided translation, then just include them as they should be. If all sprites can't make use of the provided translation -- make a better translation.

Completely agreed. Nothing wrong with including a handful, I just find it unfortunate to have to include the entire set or even a significant chunk of it.

Share this post


Link to post

I'm a little surprised there doesn't seem to be much interest in this.

I've received mixed responses privately from both port authors and modders, some say no, some say yes, some say maybe.

I thought it would've been a bit more popular. :/

Share this post


Link to post

I am not surprised. This is precisely what was to be expected, knowing how things went in the past.

Share this post


Link to post

The problem I'm seeing right now is, how do I know what lumps this is supposed to be applied to?

Share this post


Link to post

Algorithmically? It would apply to every paletted graphics lump from an archive loaded before the archive containing the PALREMAP lump (or however it's called).

Example:

[optional archive -1: your source port's support data archive]
archive 0: doom2.wad
archive 1: some gameplay mod with different guns or whatever
archive 2: adventures in newpaletteland, a bunch of maps with a new palette and a PALREMAP lump.
archive 3: paletteland DLC pack, more maps and more new graphics.

The translation table comes from archive 2, so content from archive 2 and 3 aren't reloaded; while content from archives -1, 0, and 1 are.

Share this post


Link to post
Quasar said:

The problem I'm seeing right now is, how do I know what lumps this is supposed to be applied to?

At minimum, to every paletted image in the IWAD.

Though not strictly necessary, it'd be nice if it applied to autoloaded resources too, since they're kind of a "runtime patch" more than a mod. This is so if you have the Minor Sprite Fix project or something like it loaded, you'll still get your translation even though it isn't strictly part of the IWAD.

Anything loaded manually would not get the translation.

Share this post


Link to post
Gez said:

Algorithmically? It would apply to every paletted graphics lump from an archive loaded before the archive containing the PALREMAP lump (or however it's called).

Example:

[optional archive -1: your source port's support data archive]
archive 0: doom2.wad
archive 1: some gameplay mod with different guns or whatever
archive 2: adventures in newpaletteland, a bunch of maps with a new palette and a PALREMAP lump.
archive 3: paletteland DLC pack, more maps and more new graphics.

The translation table comes from archive 2, so content from archive 2 and 3 aren't reloaded; while content from archives -1, 0, and 1 are.

Makes sense. I don't see why we couldn't support this.

Share this post


Link to post

It sounds like a great idea. Instead of a 257-byte lump, I'd lean more towards a 256-byte lump with one of two possible names to handle the "true-color passthrough" flag.

I suggest this simply because the 257-byte lump needs a special editor (or hex editor), and it contains "magic" data (it's unnecessarily complex), whereas "PALREMAP" and "TRUREMAP" (or whatever) are easy to build and understand.

One immediate use of this is to free up the 7 duplicate colors in Doom's palette, for the ability to add 7 new colors (like that blue-green everyone loves) absolutely for free.

Share this post


Link to post

Bit of a bump, but I wanted to throw my support behind the idea and also point out that it'd be useful as hell even in a single-port context. I recently nixed the idea of adding dark red ranges to a custom palette for a ZDoom map because I didn't feel like going through the hassle of converting the IWAD sprites that would inevitably be affected. I'd love to revisit the idea if given this feature.

Since both Graf and Quasar are behind it, I'd urge both to go ahead and set the trend, else nothing'll get done. Graf's format and Gez's load-order suggestions sound quite reasonable, so there's a plan. :P


[I'm less sold on the PALREMAP/TRUEREMAP split because then you've also got to define what happens if a wad includes both for whatever reason.]

Share this post


Link to post

I don't think PALREMAP needs to be anything more or less than 256 bytes. And I certainly don't think we need to be worrying about true color translations, let's just stick to 8 bit.

More than just translating assets needs to be taken into account. Namely, stuff like automap colors (should be straightforward) and sprite translation (might be more complicated).

Share this post


Link to post

Unless there's already an editor for such a binary lump in SLADE, then I think it should use the ZDoom translation specification syntax instead, which is textual and can already be parsed by all of the interested source ports (with some exceptions; EE does not support "desaturated" translations for example).

Share this post


Link to post

Less work for me so I'd be fine with that idea.

If we were to go with the binary approach, I'd suggest putting the flag byte at the end instead of the beginning. And make it optional. That way, whatever tool could be used to create them could also be used to create the player color translation tables Hexen uses.

Share this post


Link to post

Wouldn't it be far more useful in the long run for any interested software ports to convert their renderers to 32-bit, so they can display any number of differently palettized graphics, and then just have a special lump in a WAD that means "hey only draw the lumps contained in this WAD using the custom PLAYPAL/COLORMAP also in this WAD"? Or is this so obviously infeasible that no one is even mentioning it.

Share this post


Link to post

It'd be much more work because you're not just asking for having a routine that iterates through memory to replace some values with some others. You're instead looking at updating all the drawers to work in 32-bit instead of 8, and the software renderer has tons of separate drawers -- three different drawers just for columns (solid, masked, and fuzzy varieties), add at least one more if you want to handle translucency (like Boom and ZDoom do).

Share this post


Link to post
Linguica said:

Wouldn't it be far more useful in the long run for any interested software ports to convert their renderers to 32-bit, so they can display any number of differently palettized graphics, and then just have a special lump in a WAD that means "hey only draw the lumps contained in this WAD using the custom PLAYPAL/COLORMAP also in this WAD"? Or is this so obviously infeasible that no one is even mentioning it.

Absolutely going 32-bit is the right approach, ultimately. But this hack will work in the 8-bit renderers. Getting 32-bit color to work across all modern software render needs requires a lot of preparation, and a lot of work. Many non-trivial decisions need to be made.

Translucency becomes a lot slower. Using color lightmaps and deep water transmaps require new algorithms. Palette swaps (damage/radsuit/pickup) are different. Screenshot format. Do you retain 8-bit renderer support? Then you need a method to swap out renderers.

Perhaps the biggest issue is that, to take advantage of being a 32-bit renderer, the programmer will want to provide a lot of additional functionality: 32-bit textures and flats (PNG?), true-color player color remapping/text color translation.

It light of all of that, the prospect of simply adding a palette swap lump is looking better and better.

Share this post


Link to post

Exactly that. There's a true color SW renderer submission for ZDoom waiting for approval and in fact it's huge. And in order to minimize the loss of performance it makes heavy use of SSE intrinsics and even multithreaded rendering - and yet cannot keep up with 8 bit because there's 4x as much data to process.

Share this post


Link to post
Linguica said:

Wouldn't it be far more useful in the long run for any interested software ports to convert their renderers to 32-bit, so they can display any number of differently palettized graphics, and then just have a special lump in a WAD that means "hey only draw the lumps contained in this WAD using the custom PLAYPAL/COLORMAP also in this WAD"? Or is this so obviously infeasible that no one is even mentioning it.

If I'm understanding correctly, you're proposing rendering different parts of the screen with different palettes?

I think it would complicate things. Like, how to tranmap the result of two textures using different palettes overlapping? An easy solution, of course, is to calculate translucency in real time - I can already hear Graf Zahl's distant evil laughter.

Share this post


Link to post
Da Werecat said:

An easy solution, of course, is to calculate translucency in real time - I can already hear Graf Zahl's distant evil laughter.



That's exactly how it needs to be done. The currently under-review true color PR for ZDoom does precisely that.

And since you mentioned it, TRANMAP and true color are two utterly hostile concepts. This is one reason why ZDoom has never implemented this method.

Share this post


Link to post

The proposed way of having more than 256 colors on the screen seems to be aimed at preserving the paletted look of the game, even if the output is technically going to be true color. I felt it wouldn't hurt to outline that you'll probably have to make sacrifices nevertheless.

Share this post


Link to post
Linguica said:

Wouldn't it be far more useful in the long run for any interested software ports to convert their renderers to 32-bit, so they can display any number of differently palettized graphics, and then just have a special lump in a WAD that means "hey only draw the lumps contained in this WAD using the custom PLAYPAL/COLORMAP also in this WAD"? Or is this so obviously infeasible that no one is even mentioning it.


This is such a good idea we should have done it 15 years ago. Back when we started Freedoom and had in mind creating a set of hi-res texture replacements, for the at-the-time non-existent hi-res doom ports. I was sad we abandoned that goal. We should have done that too.

Share this post


Link to post
Jon said:

This is such a good idea we should have done it 15 years ago. Back when we started Freedoom and had in mind creating a set of hi-res texture replacements, for the at-the-time non-existent hi-res doom ports. I was sad we abandoned that goal. We should have done that too.

Problem is, 32-bit rendering can be a huge performance problem. 8-bit paletted-mode gives you the most bang for your buck. When you go 32-bit, you're reading a lot more data, writing a lot more data, and thrashing the cache a lot more. And translucency becomes a dog - even for your 17-3200k gaming box. Furthermore, the palette flashes (damage, rad-suit, pickups) must be calculated per-pixel, whereas in 8-bit mode, it's practically free.

All of a sudden, ,your fast gaming box is having difficulty keeping up the frame rate. At the same time, people are now creating wads that will require 32-bit mode. Ports will have to incorporate "convert to 8-bit" code to allow you to even switch back to 8-bit mode rendering. And, it'll look ugly, cause these new mods choose textures and sprites with colors that are not in the palette.

Having said all that, yeah, 32-bit mode will be nice, and should be done in a standard way, discussed by the major port authors (emphasis directed to those people you don't hear words like "standard" or "collaborate" or "cooperate" - you know who you are, and so does everyone else!) Please give everyone a chance to settle in on a common method. Thank you.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×