Baron of Hell
Register | User Profile | Member List | F.A.Q | Privacy Policy | New Blog | Search Forums | Forums Home
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Classic Doom > Source Ports > Mocha Doom TrueColor teaser...
Pages (2): [1] 2 »  
Author
All times are GMT. The time now is 07:54. Post New Thread    Post A Reply
Maes
I like big butts!


Posts: 12412
Registered: 07-06


After a long time leaving it on the side, after having worked on the HiColor branch with good results (but without officially releasing anything), I realized that splitting the codebases was a Bad Thing, because major changes became harder and harder to keep in sync.

So, harvesting the power of Java's generics I managed to bring everything together in a unified codebase with a relatively elegant solution, so that the traditional Indexed (8-bit), HiColor (15-bit RGB), and TrueColor (32-bit ARGB) versions can use the same game logic and enchancements, reusing as much code as possible, and only specializing when it's really needed.

Some comparative screenshots of E1M1:

Indexed (8-bit) mode: 163 distinct colors.

http://www4.picturepush.com/photo/a/10111302/640/10111302.png

HiColor (15-bit RGB with 32 colormaps): 270 distinct colors

http://www4.picturepush.com/photo/a/10111382/640/10111382.png

TrueColor (32-bit ARGB, using 32 colormaps): 726 distinct colors

http://www2.picturepush.com/photo/a/10111425/640/10111425.png

TrueColor (32-bit ARGB, using 256 colormaps): 1288 distinct colors

http://www2.picturepush.com/photo/a/10111450/640/10111450.png

Increasing the bit depth is not enough: the number of colormaps AND light levels (as well as the rules for how they apply at different depths) must also be changed in order to be able to see a difference.

With the HiColor mode, it's pointless to go beyond 32 colormaps, as there are only 32 possible shades of any color, and they would be indistinguishable. With 24-bit RGB, it makes sense to have more colormaps. Currently I applied my understanding of how the various lighting constants (e.g. MAXLIGHTSCALE, MAXLIGHTZ, LIGHTSCALESHIFT etc.) work, and it seems mostly OK, giving smooth and continuous gradients, though they are among the most poorly documented and obscure features.

After I replace everything in the HEAD branch (it's currently in HiColor) with the current version, I will finally release it for everybody to tinker with, as Mocha Doom v1.6 Extended Color Alpha ;-)

Sure, there are a few other enhancements...but I think I'll post some more teaser screenies for now.

E2M2: 1648 colors

http://www2.picturepush.com/photo/a/10111825/220/10111825.png

E2M1: 548 colors (meh...poor decor ;-)

http://www1.picturepush.com/photo/a/10111889/220/10111889.png

Unholy Cathedral: 1115 colors

http://www3.picturepush.com/photo/a/10111946/640/10111946.png

MAP01 from DOOM II: 1854 colors

http://www4.picturepush.com/photo/a/10112047/220/10112047.png

1300+ colors:

http://www1.picturepush.com/photo/a/10112084/640/10112084.png

For anyone wondering, my method for achieving extended color depth differs from _bruce_'s chocolate doom branch: I still use colormaps for applying color and distance effects, with a few tweaks such as real-time palette manipulation (palette and colormaps effects are preserved, though there are a few glitchs), and the drawing functions are very close to the original 8-bit ones. The colormaps however now carry RGB values instead of 8-bit indices, and are appropriately sized (e.g. ints for TrueColor, shorts for HiColor).

This allows very good performance, with almost no speed penalty. E.g. nuts.wad timedemo @ 900 x 600 fullscreen (full resolution, not screen tripling!)

Indexed: timed 4048 gametics in 3068 realtics = 46,179922 frames per second

HiColor: timed 4048 gametics in 3256 realtics = 43,513514 frames per second

TrueColor: timed 4048 gametics in 3280 realtics = 43,195122 frames per second

Per-pixel processing and HSV-RGB conversions are avoided, though still possible. It's also possible to apply individual colormap effects, modifying colormaps on-the-fly, caching frequently used ones etc.

Last edited by Maes on 09-24-12 at 17:51

Old Post 09-24-12 17:35 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Gez
Why don't I have a custom title by now?!


Posts: 11072
Registered: 07-07


Can your colormap-based method cope with unpaletted graphics? Or does it makes Mocha still tied to the palette for source images?

Anyway, looking very smooth.

Old Post 09-24-12 18:08 #
Gez is online now Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06



Gez said:
Can your colormap-based method cope with unpaletted graphics?



Yes and no ;-)

The Yes part: I genericized two aspects of graphics resources, whenever possible, and most classes look like e.g. Renderer<T,V>, where T is the type of underlying graphics, and V is the type of VIDEO OUTPUT to produce. Also, in theory, nobody prevents writing column functions that produce their visuals in a non-palletized way (e.g. drawing hicolor resources, using extended patch_t and column_t formats or procedurally).

The real difficulty would be writing code that handles mixed types cleanly. E.g. a palettized sprite will have to be drawn using colormaps and the usual light calculations, but a RGB one would need totally different code, and applying effects to it would require full pixel-based processing. I might give it a try with something relatively isolated e.g. a HiColor or TrueColor sky.

The No part: currently, <T> is only byte[] in all implemented classes, while <V> can obviously be byte[], short[] and int[], to produce the Indexed, HiColor and TrueColor modes. Also, some classes are currently only implemented for byte[] type of resources, so in essence everything still relies on resources being in the good old Doom format.

The resource handling system is Doom all the way: it expects patch_t for resources, that break up into column_t's, which in the end result in a bunch of byte[]. It's very easy to extend stuff so that e.g. column_t's contain short[] or int[], and modify drawing functions so that they don't use colormaps. The problem is that:


  • You would need to pre-convert palettized resources to the new formats before using them, even if on-the-fly. This effectively "freezes" their colors.
  • You lose a lot of optimizations (e.g. super-fast lighting and tinting effects using palettized colormaps are no longer possible).
  • You might regain some speed by using mixed handling (e.g. using colormaps AND direct drawing), but that will make the code a mess to maintain.
  • There are no standard disk formats for resources in extended color formats. It would be a pity to put work into something using your own custom formats and assumptions, only to see that nobody uses them.


Still, the idea is tempting *scratchchin*

Last edited by Maes on 09-24-12 at 18:39

Old Post 09-24-12 18:25 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
_bruce_
Senior Member


Posts: 1290
Registered: 11-07


Awesome job.
You're a great engineer Maes.

Old Post 09-25-12 20:34 #
_bruce_ is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
Siggi
Member


Posts: 388
Registered: 10-04


I like all the 32bit-ness that's been happening recently.

I think there's a bug with your lighting, it's most obvious with your MAP01 screenshot. It's way too bright. IIRC, the problem there is that you're using the current high resolutions height instead of 200 as Doom would. It looks like the same bug PrBoom 2.02 has. I'm pretty sure poking around the PrBoom source will help find the solution I'm thinking of. Had the same issue with my pet source port.

edit: i mean the spans not the columns

Last edited by Siggi on 09-25-12 at 23:28

Old Post 09-25-12 21:54 #
Siggi is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06



Siggi said:
Stuff about lighting


I admit I only have an empirical, superficial knowledge of the lighting system, and if I managed to calibrate certain of the lighting variables for variable bit depths (the default "bit depth" is 5, for 32 color maps) it was through trial and error or educated guesses.

In rr.LightsAndColors.java: http://pastebin.com/RdW6tabh you can see how I made setting up certain variables dependent on the bit depth, and I commented them according to what I think their use is.

The gory bit is this one:

code:
LIGHTLEVELS = 1<<LBITS; LIGHTSEGSHIFT = 8-LBITS; NUMCOLORMAPS= LIGHTLEVELS; MAXLIGHTSCALE = LIGHTLEVELS+LIGHTLEVELS/2; LIGHTSCALESHIFT = 17-LBITS; LIGHTZSHIFT=25-LBITS;


Where LBITS=5 for 32 colormaps, 6 for 64, 7 for 128, and 8 for 256.

The only one I haven't come up with a precise rule is this one:

code:
/** This one seems arbitrary. Will auto-fit to 128 possible levels? */ public static final MAXLIGHTZ=256;


which has a default value of 128 with 32 colormaps. I just bumped it up to 256 (which seems to work fine even when using 8-bit modes...)

If anyone knows precisely how these values affect the lighting effect, I'd appreciate some insight...

Edit: prBoom+ uses 320 as a screen WIDTH constant in R_InitLightTables, independent of resolution. I made the same change, and now the screenshots look like this (TrueColor mode):

http://www1.picturepush.com/photo/a/10197914/640/10197914.png

http://www1.picturepush.com/photo/a/10197934/640/10197934.png

In Indexed mode:

http://www3.picturepush.com/photo/a/10197946/640/10197946.png

I've got to admit, E1M1 looks better, too, and the color count was bumped up to 1552...

http://www2.picturepush.com/photo/a/10198410/640/10198410.png

Last edited by Maes on 09-26-12 at 10:28

Old Post 09-26-12 09:38 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Dragonsbrethren
Senior Member


Posts: 2409
Registered: 03-09


Yeah, that lighting looks much better.

Old Post 09-26-12 17:11 #
Dragonsbrethren is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
AlexMax
Senior Member


Posts: 1114
Registered: 01-03


Would it be possible to recreate the 'banded' light dropoff that happens in 8-bit modes in higher color modes? I love the additional color depth, but my eyes are simply too used to the banding and the smooth light falloff looks...wrong.

Old Post 09-27-12 02:08 #
AlexMax is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
lucius
Junior Member


Posts: 138
Registered: 04-08



AlexMax said:
Would it be possible to recreate the 'banded' light dropoff that happens in 8-bit modes in higher color modes? I love the additional color depth, but my eyes are simply too used to the banding and the smooth light falloff looks...wrong.
The number of colormaps used will affect the banding. Using 32 light levels with true color colormaps should give you the banding without the color distortion found in 8 bit mode.

Old Post 09-27-12 08:03 #
lucius is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
esselfortium
A Major Doomworld Concern


Posts: 6522
Registered: 01-02


Something still seems off. The lighting on the floor looks significantly darker than the lighting on the walls, and Doom's fake-contrast brightening/darkening of walls isn't enough to account for it.

Old Post 09-27-12 08:33 #
esselfortium is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06



lucius said:
The number of colormaps used will affect the banding. Using 32 light levels with true color colormaps should give you the banding without the color distortion found in 8 bit mode.


That is correct. In fact, unless more colormaps and light levels are used, banding is still very noticeable. For technical reasons, the number of light levels can be only a power of two.

As for the light levels being off...that's why I posted the light costants and code before, in case someone knows better than me how they are supposed to scale up, and whether there are other gotchas like the initialization of floor lights being based on using a constant screen width of 320 pixels, and not of the actual screen width.

However, a very obscure point is the value of the MAXLIGHTSCALE constant. In vanilla doom, its value is 48, so I would be tempted to derive that MAXLIGHTSCALE = NUMLIGHTLEVELS + NUMLIGHTLEVELS/2 formula (if based on 32 levels) or 3*NUMLIGHTLEVELS (if the constant was set to 16).

BTW, a side effect of using more color depth is that everything tends to appear darker simply because the colors now fade like they were supposed to, rather than being remapped to their "closest" colors in the palette. Of course, this might alter the appearance of certain stuff. The HiColor mode is not all that better, because using just 32 levels per color component gives very little improvement in overall appearance, and fades are nearly as coarse as with the palettized version.

Last edited by Maes on 09-27-12 at 10:07

Old Post 09-27-12 10:01 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06


Do these look more natural? I reasoned that since the original hardcoded values supplied with the LinuxDoom code were probably calibrated to work with 16 colormaps, I had to change some assumptions about the shifts etc.

http://www3.picturepush.com/photo/a/10263366/640/10263366.png


http://www3.picturepush.com/photo/a/10263376/640/10263376.png

All I did was change those:

code:
LIGHTLEVELS = 1<<LBITS; LIGHTSEGSHIFT = 8-LBITS; NUMCOLORMAPS= LIGHTLEVELS; MAXLIGHTSCALE = 3*LIGHTLEVELS; LIGHTSCALESHIFT = 17-LBITS; LIGHTZSHIFT=25-LBITS;


into these:

code:
LIGHTLEVELS = 1<<LBITS; LIGHTSEGSHIFT = 8-LBITS; NUMCOLORMAPS= LIGHTLEVELS; MAXLIGHTSCALE = 3*LIGHTLEVELS; LIGHTSCALESHIFT = 16-LBITS; LIGHTZSHIFT=24-LBITS;


Substituting 4 for LBITS, gives 16 Light levels, MAXLIGHTSCALE=48, ligthscaleshift=12, lightzshift=20 (default LinuxDoom values).

Edit: by tweaking the above so that
code:
LIGHTSCALESHIFT = 17-LBITS;


again, as would be normal if 32 light levels were standard, instead of 16, results in the following:

http://www1.picturepush.com/photo/a/10263714/640/10263714.png

http://www1.picturepush.com/photo/a/10263729/640/10263729.png

Old Post 09-27-12 15:36 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06


I think I finally nailed it: I needed to introduce the LIGHTBRIGHT constant from prBoom as well as use the 320 multiplier globally, which seems to have restored relative brightneses to correct levels.

I "formulated" it as LIGHTBRIGHT=LBITS>>2; (so it's 1 for 16 light levels, 2 for 32 etc.).

Now there's another problem: the MAXLIGHTZ constant must be scaled to fit, otherwise floors become too bright. Values of 256 work OK, but values of 512 and 1024 seem to squeeze a bit more colors.

MAXLIGHTZ 256: 1829 colors

http://www4.picturepush.com/photo/a/10267237/640/10267237.png

MAXLIGHTZ 512: 2307 colors
http://www5.picturepush.com/photo/a/10267308/640/10267308.png

MAXLIGHTZ 1024: 2492 colors

http://www3.picturepush.com/photo/a/10267136/640/10267136.png

Going beyond 1024 levels doesn't seem so helpful, and it's probably a cache killer.

A common "feature" of all extended color modes are significant hue alterations, compared to the 8-bit renderer: most browns tend to become lighter/greyish or more yellow. To be fair, however, it's the 8-bit renderer that's actually altering how Doom is supposed to look ;-)

Old Post 09-27-12 16:56 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
wesleyjohnson
Senior Member


Posts: 1002
Registered: 04-09


I have been working with 32bit draw on the DoomLegacy Fog and have observed the same color shifts. A brown fog will turn yellow at some alpha values. I suspect it is because of accumulated rounding errors in the RGB calculations. Sometimes the round-offs are going to align to move the hue significantly.

The doom palettes and colormaps were tuned to the closest color, using methods and human intervention beyond what our fast 8 bit math can produce. Would have to use a couple bits of fraction in the RGB components to fight this.

I try to use the colormaps when possible, and translate to 32bit color right before drawing or transparency math.
The 32bit RGB appears better where tranparencies are involved, because the transparency map can only return a palette color, and many of them are not very close.

The Doom colors seem to have enough intensity range in the palette to look good at all the Doom intensities.
RGB colormaps is something that I need to eventually finish in DoomLegacy. I have also considered expanding the number of light intensity levels in the colormaps. But right now the 32bit draw looks fairly good, even at the usual number of light level colormaps.

Have to look at how the Gamma settings affect the colormaps.
That cannot be duplicated reasonably in 32bit RGB math (the curves are non-linear). I have 32 level Gamma controls with 3 non-linear control variables, so I would not try that on DoomLegacy. If you have replaced your Gamma table generation with some linear math, then you are using a linear gamma approximation, which will have significant errors everywhere except two points. It will also be tuned for your monitor and others will not be able to tune it to their monitor. This will be especially bad for Mac and LCD screens which have different gamma from PC CRT displays. I mention this because some of your color problems sound similar to what happens when the gamma curve does not match the monitor.

I have seen the generic draw generator in another port, and I would not do it that way. I preferred to customize the math of each draw routine individually. The RGB math for 16 bit looks nothing like the RGB math for 32 bit.

Last edited by wesleyjohnson on 09-27-12 at 20:37

Old Post 09-27-12 19:45 #
wesleyjohnson is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06



wesleyjohnson said:
The doom palettes and colormaps were tuned to the closest color, using methods and human intervention beyond what our fast 8 bit math can produce.


I beg to differ (check out dcolors.c). The methods ID used to produce the COLORMAP lump (and the unused COLORS15 and COLORS12) are precisely the sort of "fast 8-bit math" you describe.

The procedure itself (at least for the COLORS15 lump) is sound from both a mathematical standpoind, and from a color theory one: they start from an initial palette (the only one which might have been manually tuned) have 32 light levels, and they shift each of the color components down by one unit until everything turns pitch black. This works just as good as possible with 15 or 12 bit color, accordingly, and scales well to 24-bit one, if one takes the time to extend it (as I did).

The problem is that the 8-bit "colormaps" used by vanilla doom are simply remappings into The One True Doom Palette palette, so even the best "RGB matching" you can do, will be a compromise (of course, that doesn't mean it can't be tweaked e.g. to prefer closest hue than closest luminosity, but that's NOT done in dmutils). The HiColor and TrueColor modes are actually showing what the light transitions should look like if proper color depth was obtainable. Certain light transitions look the way they do in vanilla because they could not be made to look any other way (and some, especially at low light levels, are quite ugly, e.g. orange becoming purple).

The HiColor RGB555 mode used in the Alphas, however, is not so much better looking because of limited shades per color, and IMO would not be worth it, not even with RGB565. With true color and more light levels, however, it's totally different story.

And, surprise surprise, the more you crank the light levels up, the more it starts to look like hardware rendering... the ideal would be to find a compromise between the "hardwarey" smoothed look, and the gritty, banded look of the 8-bit software renderer.

As for gamma tables, all screenshots are taken at gamma level 0 (unmodified). For producing gamma-corrected palette and colormaps effects, I've been using the gamma curves included in the original Doom (I think they are 5 levels, including "level zero", with separate curves for R,G,B).

Last edited by Maes on 09-27-12 at 22:14

Old Post 09-27-12 21:21 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Gez
Why don't I have a custom title by now?!


Posts: 11072
Registered: 07-07



Maes said:
I beg to differ (check out dcolors.c). The methods ID used to produce the COLORMAP lump (and the unused COLORS15 and COLORS12) are precisely the sort of "fast 8-bit math" you describe.


Besides, I've found out that it actually looks better than much more complicated maths. Man, look at that CIEDE2000 function; she's a beaut, she is. :)

Anyway. For fun I made it so that you can change which color matching algorithm is used in SLADE 3. Look at the Graphics->Colorimetry tab in the preferences; more advanced gobbledygook than most people will ever need to try to begin to understand. Without touching anything else, change the color matching method to something, then generate a colormap from the playpal. Save, and try in Doom.

Note: if you use the "Carmack's Typo" greyscale luminance preset, and the RGB (integer) color matching function, then the COLORMAP you'll generate from Doom's PLAYPAL will be exactly identical to Doom's own COLORMAP.

If you use the HSL colormatching method, you'll start to get interesting results. And with the CIE colormatching methods, you'll obtain a COLORMAP that, when viewed as an image, seems closer to what it should be (reds remain red, etc.) but actually look weird in-game. (I recommend Doom II MAP28: The Spirit World for testing colormaps).

If you tweak the HSL factors, you might eventually find values for which HSL colormatching gives good results; however I wouldn't bother too much. RGB works best with the Doom palette.

Old Post 09-27-12 22:01 #
Gez is online now Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
DaniJ
Senior Member


Posts: 2081
Registered: 08-03


Heres an idea:

As hue shift in DOOM's lighting model is borne of the colormap remapping, it should be possible to do something similar in hi-color modes. Preprocess the colormaps to compute HSV shift vectors over a finite range of light levels. Model the results along the lines of a spline matching a signal and here is a function which could be incorporated dynamically in the column renderer.

Personally I think that replicating the hue shift is desirable because it is intrinsic to the visual style of the original game. Although this only came about through a technical limitation, DOOM without the hue shift looks rather sterile to me.

Old Post 09-27-12 22:10 #
DaniJ is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Siggi
Member


Posts: 388
Registered: 10-04


I still need to look through what I did to fix the lighting weirdness with my port and go through everything Maes has said here, but I think this may help as far as comparisons go.

My current mystery is why the pistol is so dark, heh.

Old Post 09-28-12 01:21 #
Siggi is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06



DaniJ said:
emulating color shifts


No need to actually do that with HiColor RBG555 modes: with just 5 bits per color and 32 colormaps, they look nearly as gritty as the 8-bit ones, if not worse in a few cases, where palette "solarization" is very evident, and there are actually colors in the Doom palette that cannot be expressed 100% accurately in HiColor mode (there was even a 12-bit RGB mode planned...tried that, and it looked amusingly bad).

However, it should be possible to simply do a color-to-color interpolation between existing colormaps, and blow their number up, giving intermediate hues but preserving the (desirable?) 8-bit hue distortions. This can be done relatively cheaply on startup, no need to have modified drawing functions.

Old Post 09-28-12 08:15 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
DaniJ
Senior Member


Posts: 2081
Registered: 08-03


I wasn't referring to the RGB555 mode specifically, what I was thinking of would be applicable to all modes, regardless of color precision.

I'm guessing you mean simply doing a Gaussian blurring of the COLORMAP to yield a greater number of colors while retaining the existing darkening range separations and hue shift aberration "forms"? If so then yeah, that sounds like what I'd like to see.

Though I still think the use of COLORMAP luts in a renderer like yours is something of a "cheat" - I'd love to see a version of the DOOM lighting model implemented purely with infinite resolution functions for each term.

Old Post 09-28-12 09:46 #
DaniJ is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06



DaniJ said:
Though I still think the use of COLORMAP luts in a renderer like yours is something of a "cheat" - I'd love to see a version of the DOOM lighting model implemented purely with infinite resolution functions for each term.


_bruce_'s chocolate doom truecolor branch does just that -full framebuffer processing in HSV colorspace, resources pre-converted to true-color, fully calculated lighting etc.

The price to pay for that flexibility, however, is performance, which I wasn't willing to sacrifice, and the need to rewrite most of the lighting system.

I was more interested in seeing how well the colormap model (in particular the one used in the Alphas) could work if given more bit depth to play with. Essentially, such modes are "super-indexed" modes, giving more flavour to 8-bit data, with relatively little processing overhead, and minor changes in the code.

Another approach I have seen was the one used in early DelphiDoom. That one, if I understood correctly, actually dynamically interpolated light levels between z-planes and even on walls, thus implementing your "infinite resolution functions" in its own way. Did it work? Yes. However it looked blurry like an OpwnGL linear filter, and once again the price to pay was performance: it struggled to even attain vanilla-like framerate on vanilla maps. With the colormap approach, the slowdown between indexed and hicolor/truecolor is less than 10%.

Edit: heh, reading that old thread reminded me how sooner or later all such "extended color depth doom" projects stumble upon the following dilemmas/dead-ends:


  • Some people won't like the "smoothed" look.
  • Color extension is only done at the video output (canvas) end, not at resources side.
  • Supporting graphic resources with more than 8-bit color is a separate matter altogether, and would create extremely bloated codebases, or call for separate branches.
  • If the colormaps lighting model is abandoned, there's a steep price to pay in terms of performance and code reuse. Steep as in "you won't even get vanilla framerate" steep.


These aren't exactly unsolvable, but it's not easy nor worth it to do so. E.g. why bother with using exclusively 32-bit resources, if that means losing the fast colormap lighting and with no such resources to begin with? Drawing a truecolor sky (WHICH HAS NO LIGHTING EFFECTS APPLIED TO IT) is one thing, extending this to ALL sprites, flats, etc. AND preserving the lighting effects is another.

Last edited by Maes on 09-28-12 at 11:53

Old Post 09-28-12 11:34 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
wesleyjohnson
Senior Member


Posts: 1002
Registered: 04-09


Sorry, I was not trying for historic accuracy in my comments, but pointing out why precalculated tables have such an advantage over run-time RGB math. This is long-winded because there are other viewers of this forum than just experts and I tried to give details.

The hand tuning in the palette will not have left visible traces, and I really don't care about whether they actually did this or not (it does not matter for this discussion). Some palette color choices will change hue when converted to darker colors using integer math.
This is because of unequal proportions lost when low order bits are lost. There are fewer choices of hue in the darker colors. To get a good palette for a game, select the dark colors first, and then multiply them up to the bright colors for your palette. This gives palette colors that will retain their hue when darkened. It does depend upon exactly what operation is used to darken a color, and that can be hand-tuned too. This hand-tuning of the palette is necessary given the integer RGB representation.

The hue error (hue shift) is NOT identical to the round-off error of a single RGB component. It is due to the difference in round-off between the RGB components. Using simple math with no round-off then there is a larger truncation error instead, but with very similar hue error.
This effect is made worse in Doom because it is synchronous over a large area. Due to a limited palette, when a worst case error occurs for one color combination, then it occurs identically wherever the same colors combine, over the whole texture or sprite.
This can be minimized by using an RGB framebuffer, because any individual pixel differences are preserved (the color quantization of a palette based framebuffer makes the effect far worse). However, because of the source color quantization, there is limited opportunity to introduce differences. It may actually help to introduce a deliberate random least-sig-bit (dither).

The gamma calculations that create the gamma table are all done in float in DoomLegacy (I did not look at other ports). The doom palette does a lookup in the gamma table to create the video card palette. There is no reason to use integer in gamma table generation as it would introduce even more error in the base palette.
Overall, this does not create much error.

All the colormaps are loaded, and are Doom color to Doom color translations. This is subject to color quantization errors. If the calculations that created the colormaps are very simple, then float would not be needed. The round-off error of float-to-int conversion is about the same as one integer operation. Trying to replace any of the colormap lookups with RGB math can eliminate the color quantization error, but will introduce round-off error instead, in proportion to how many adds and multiplies and shifts are used.

That leaves the RGB operations as a source of hue error, which is where all this is leads. The concepts of accuracy, precision, significant digits, complicate the whole issue of how much error is in a RGB calculation. I trust interval arithmetic the most. The others are estimates. Because RGB is not subject to measurement errors, the statistic basic equations for combining accuracies do not apply.

To shorten this, let UE be the uncertainity error. If the number of bits in intermediate values is not constant then it would be preferable to track the number of significant bits.
I tried to verify this stuff from my numerical analysis notebooks, but they are not readily available. I expect there will be alternative error analysis argued. This is a very rough estimate.
A value from a table has at least a 1/2 bit UE.
The result of adding two values is 1 bit UE.
A double-length multiply result does not itself have additional UE, but shifting off the low order fractional bits does. A division is complicated but if you are looking for speed you won't be using any. Multiply by a constant, multiplies the possible error by the constant.

Doing one transparency RGB calculation using integer math, is 1 add, and a shift (minimum). This gives 2 bit UE estimate leaving 6 bits of accurate color.
The hue shift occurs because of the independence in the actual error direction in each component.
Even with 32bit color, there are only 8 bits in each component. In Doom this is actually only 7 bits because it mostly uses the darker colors (Using INTEGER, darker colors have less available range, and less tolerance for UE, than bright colors do).

There are only two apparent ways to prevent this, use FLOAT or FIXED POINT to do all RGB component math and round the result back to INTEGER, or minimize the number of integer operations on RGB components, preferably to fewer operations than 1/4 of the bits available.

Old Post 09-29-12 23:32 #
wesleyjohnson is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06


OK, but all of this is totally irrelevant because it's a not a matter of doing calculations with less-than-ideal accuracy. Even doing them with the best accuracy possible would not change a thing because you only have 256 colors to simulate the darkening of those very same 256 colors, and then the "darkening" of those "darkened" 256 colors etc. until you're left with practically one or two colors for the darkest colormaps (in vanilla Doom, of course).

YES, you could compute a certain black to actually be #010001 instead of #000000, but NO, it wouldn't change a thing because color #010001 is not on the initial palette, so you will have to compromise, and pick #000000 anyway.

Bottom line: at each darkening step you WILL lose distinct colors, and no amount of optimizing the initial palette or calculating hues with arbitrary precision will fix that.

Old Post 09-29-12 23:46 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
DaniJ
Senior Member


Posts: 2081
Registered: 08-03



Maes said:


_bruce_'s chocolate doom truecolor branch does just that -full framebuffer processing in HSV colorspace, resources pre-converted to true-color, fully calculated lighting etc.

The price to pay for that flexibility, however, is performance, which I wasn't willing to sacrifice, and the need to rewrite most of the lighting system.


Thanks for the tip. That does sound very much like what I'm interested in seeing. Performance isn't really a concern for me, I'm more interested in this from engineerial and mathematical perspectives.

Old Post 10-01-12 00:48 #
DaniJ is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
wesleyjohnson
Senior Member


Posts: 1002
Registered: 04-09


No ... you don't mix fast 32bit RGB math, and then do closest palette color search. You only do fast 32bit RGB math when the result gets written directly to the video buffer, thus avoiding color quantization errors entirely.

The analysis shows:
- Before draw-time use FLOAT or FIXED to make any tables, and then round or truncate color components to integer. This does not involve a closest color search. It writes RGB results to tables or to the video card palette.
- During draw, minimize the operations (for speed and because fast integer RGB math adds error). These would be transparency blends of two RGB component colors (alternative to transparency tables). Blend of 50% transparency is easy and small error. Blend of arbitrary alpha is more expensive, slower, and more error, and cannot be done by small tables.
- How to minimize color shift, which is to carefully pick your starting palette. When the program applies gamma to the loaded Doom palette it calculates and writes to the video card the actual RGB values for every palette color by running the Doom palette through a non-linear gamma equation (by gamma table lookup).
Here you have the opportunity to adjust the bright colors within the palette to be multiples of the darker colors such that the darkening operation does not give a hue shift. The darker colors do not have enough significant bits to tolerate adjustment,
but the bright colors do.

I did a better numerical analysis and got more accurate error bounds.
A transparency calculation in RGB 8 bit components using int registers
only adds 1/2 bit error to any error in the tables, total 1 bit error.
But doing it with mask/shift/add (R1>>1 + R2>>1) gives 2 bit error.

WARNING MATH FOLLOWS:

I( <expr> )
: is the interval of the fully accurate values when the <expr>
is executed under the conditions of the context

* Conversion of component to integer by rounding
I(R) = [R-0.5 .. R+0.49999...] ~= R + [ -0.5 .. 0.5 ]

* Conversion of component to integer by truncation
I(R) = [R-0.0 .. R+0.99999...] ~= R + [ 0 .. 1.0 ]

* Addition in general
R1, R2 : R + [e1 .. e2]
I(R1+R2) = R1 + [e1 .. e2] + R2 + [e1 .. e2]
= R1+R2 + [(e1+e1) .. (e2+e2)]
= R1+R2 + [2*e1 .. 2*e2]

I(R1+R2+R3) = R1+R2+R3 + [3*e1 .. 3*e2]

I(R1+B2) = R1+B2 + [er1+eb1 .. er2+eb2]

* Addition of rounded integer components
R1, R2 : R + [-0.5 .. +0.5]
I(R1+R2) = R1 + [-0.5 .. +0.5] + R2 + [-0.5 .. +0.5]
= R1+R2 + [-1.0 .. +1.0]

* Addition of truncated integer components
R1, R2 : R + [0.0 .. 1.0]
I(R1+R2) = R1 + [0.0 .. 1.0] + R2 + [0.0 .. 1.0]
= R1+R2 + [0.0 .. 2.0]

* Multiplication by constant, (shift left)
I(R1 * c) = R1*c + [c*e1 .. c*e2]

* Shift right, losing LSB bits by truncation
I(R1 >> 1) = INT(R1>>1) + [ (e1/2) .. ((e2/2) + 0.5) ]

I(R1 >> n) = INT(R1>>n) + [ (e1/n) .. ((e2/n) + 1.0 - (0.5**n)) ]

* Average of two components
I((R1 + R2)>>1) = (R1 + [e1 .. e2] + R2 + [e1 .. e2]) >> 1
= (R1+R2)>>1 + [(e1+e1)/2 .. ((e2+e2)/2 + 0.5)]

* Average of two rounded components
(This requires loading components to a longer register)
I((R1 + R2)>>1) = (R1 + [-0.5 .. 0.5] + R2 + [-0.5 .. 0.5]) >>1
= (R1+R2)>>1 + [-0.5 .. 1.0]

* Average of two truncated components
(This requires loading components to a longer register)
I((R1 + R2)>>1) = (R1 + [0 .. 1.0] + R2 + [0 .. 1.0]) >>1
= (R1+R2)>>1 + [0 .. 1.5]

* Average of two components, confined within 8 bits (all intermediate results must fit within 8 bits)
(This is used for mask/shift/add math on 15bit and 16bit RGB, without separating components. When only 5 bit components are available the error band is 6.25% of full value (4.7% worst deviation). )
I( R1>>1 + R2>>1 ) = (R1 + [-0.5 .. +0.5])>>1 + (R2 + [-0.5 .. +0.5])>>1
= (R1>>1 + [-0.25 .. 0.75]) + (R2>>1 + [-0.25 .. 0.75])
= (R1>>1 + R2>>1) + [-0.5 .. 1.5]

Last edited by wesleyjohnson on 10-01-12 at 21:03

Old Post 10-01-12 20:40 #
wesleyjohnson is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06


OK, so you suggest formulas for doing (relatively expensive) color blends in RGB space. If you could come with an acceptable (fast) substitute for the BLURRY_MAP effect (which would be "half-brite" according to id), that would be something I'm interested in. For HiColor mode, I found the following method, which is one of the few framebuffer processing effects I do (with notes on each effect):

code:
private final short fuzzMix(short rgb){ // super-fast half-brite trick // 3DEF and >> 1: ok hue, but too dark // 7BDE, no shift: good compromise // 739C, no shift: results in too obvious tinting. return (short) (rgb&0x7BDE); }


I've yet to come up with a good equivalent for it in RGB space, though.

As for the Doom palettes, the problem is in the method used to choose "most similar" colors during precomputation: they use a simple nearest (RGB) neighbor method using (squared) euclidean distance stored in a long register (check dcolors.c).

Now, if colors were picked out primarily because of their hue and secondarily because of their luminosity (using a two-phase or two-keyed sort), -maybe- some hue errors would be more limited, but only to be replaced with luminosity errors and non-linear appearance (some things would be lighter, others darker).

And of course those precomputed tables are only made assuming the default (gamma 0) level of the palette, which is simply a straight curve mapping i: [0,255] to j: [0,255] 0:255. Of course there should be different colormaps for different gamma (and thus overall lightness, but who ever did that?

Old Post 10-02-12 08:51 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Gez
Why don't I have a custom title by now?!


Posts: 11072
Registered: 07-07



Maes said:
As for the Doom palettes, the problem is in the method used to choose "most similar" colors during precomputation: they use a simple nearest (RGB) neighbor method using (squared) euclidean distance stored in a long register (check dcolors.c).

Now, if colors were picked out primarily because of their hue and secondarily because of their luminosity (using a two-phase or two-keyed sort), -maybe- some hue errors would be more limited, but only to be replaced with luminosity errors and non-linear appearance (some things would be lighter, others darker).



Hey, get SLADE 3, tweak the colorimetry to your heart's content, and generate some colormaps from the playpal.

I actually haven't found a colormap more convincing in actual use than the RGB method.

Old Post 10-02-12 11:02 #
Gez is online now Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
wesleyjohnson
Senior Member


Posts: 1002
Registered: 04-09


Yes, hue fixes would trade-off luminosity errors. But only adjusting bright colors you only need to adjust by a few bits, where it should be in opposite directions for two components, keeping the luminosity close to the original.

I think the ports all apply gamma to the palette in the same way, which came from vanilla Doom. They load the Doom palette and then apply a gamma curve to it, and then load that to the video card palette.
All lookup tables generate indexes which reference the palette loaded into the video card.
There is only one set of tables, even for DoomLegacy which has many more than the usual 5 gamma settings.
The ports that use GL or other non-palette drawers would imitate those effects as they still have a gamma control.

Don't know BLURRY_MAP effect, probably because many names have changed in my port. Our invisible monster mostly uses a semi-transparent effect because it is more GL compatible.
I did make a blur effect, but it was not very good, and I do not remember how right now.
In DoomLegacy have most all the RGB effects you would need, but they cover page after page of code. A bit much to put here.

From memory:

50% transparent suitable for 24 bit and 32 bit RGB
uses structure to access components

pixel32_t * dest32 = (pixel32_t*) dest;
while( count-- )
{
pixel32_t t2 = <some texture access function>;
uint16_t r1, b1, g1;
r1 = dest32->u32.r;
dest32->pix32.u32.r = (r1 + t2.u32.r) >> 1
g1 = dest32->u32.g;
dest32->pix32.u32.g = (g1 + t2.u32.g) >> 1
b1 = dest32->u32.b;
dest32->pix32.u32.b = (b1 + t2.u32.b) >> 1
}


50% transparent in 15 bit RGB (and with different masks 16 bit RGB)
const mask_r_b = 0x7C1F;
const mask_g = 0x03E0;
const mask_1 = 0x0421;
const mask_N1 = ~ mask_1;

while( count-- )
{
// 50% transparent on all RGB components in place in 15 bit RGB
// mask LSB to 0, then divide by 2, then add
uint16_t t2 = ((<some texture access function> ) & mask_N1) >> 1;
uint16_t t1 = ((*dest) & mask_N1) >> 1;
(*dest) = t1 + t2;
}

Old Post 10-04-12 05:12 #
wesleyjohnson is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
Maes
I like big butts!


Posts: 12412
Registered: 07-06


The BLURRY_MAP effect just reads pixels at fixed positions to the left and right of the target directly from the framebuffer according to a L/R pattern (chosen by fuzzoffset and a table), and uses a relatively dark colormap (#6) for rendering them. This "pixel picking" is what gives the sparkling invisibility effect.

It's possible to replicate even with non-indexed color processing, but not necessarily faster than proper transparency.

Old Post 10-04-12 08:57 #
Maes is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
wesleyjohnson
Senior Member


Posts: 1002
Registered: 04-09


I do not think that the blurry effect for invisible monsters can be done any better than the offset table.
Just copy your 50% or 25% transparent draw code and add the table offset.
For speed avoid any IF branches. One IF branch can be worse than two or three operations.

// fuzztable has entries from
// { -sizeof(pixel32_t), 0, +sizeof(pixel32_t) }
// To avoid the operation: fuzzindex & 0x1F, can make the table 256 entries, and use a uint8_t for fuzzindex. Because its execution is overlapped with others and is not in the critical path, it would not speedup much.

pixel32_t * dest32 = (pixel32_t*) dest;

while( count-- )
{
register uint16_t r1, b1, g1;
register pixel32_t t2 = <some texture access function>;
register pixel32_t * d32b = dest32 + fuzztable32[ fuzzindex++ & 0x1F ]; // 32 entry table
r1 = d32b->u32.r;
d32b->pix32.u32.r = (r1 + t2.u32.r) >> 1
g1 = d32b->u32.g;
d32b->pix32.u32.g = (g1 + t2.u32.g) >> 1
b1 = d32b->u32.b;
d32b->pix32.u32.b = (b1 + t2.u32.b) >> 1
dest32 += vid.ybytes; // num bytes in a vid line
}

Last edited by wesleyjohnson on 10-04-12 at 21:57

Old Post 10-04-12 21:52 #
wesleyjohnson is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
All times are GMT. The time now is 07:54. Post New Thread    Post A Reply
Pages (2): [1] 2 »  
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Classic Doom > Source Ports > Mocha Doom TrueColor teaser...

Show Printable Version | Email this Page | Subscribe to this Thread

 

Forum Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is OFF
vB code is ON
Smilies are OFF
[IMG] code is ON
 

< Contact Us - Doomworld >

Powered by: vBulletin Version 2.2.5
Copyright ©2000, 2001, Jelsoft Enterprises Limited.