k8vavoom: no good thing ever dies!

[jval]

1 hour ago, Gibbon said:

 

Hope Ketmar doesn't mind me putting it here:

https://github.com/atsb/source-port-builds/releases/download/1.0/k8vavoom-git-master_win64.zip

 

Check the source port builds thread.  Windows 11, MinGW 64bit.  The only code changes were:

 

1. CMake was altered to remove the -mtune and -march as those were bleeding into the Windows CMake and flagging my CPU as non-64bit

2. All errors relating to "unsupported etc.." were removed to allow the build to create the pk3 files and the executable

3. Link.txt for CMake was altered to hardcode my path for SDL2 (no pkgconfig)

4. basev was put into the same directory as the binary as on Windows, it doesn't look in shared.

 

Plays nicely, saves and loads, etc...

 

The size of the binary files is huge, do they contain debug info? Can we have a "release" build?

[Gibbon]

18 minutes ago, jval said:

 

The size of the binary files is huge, do they contain debug info? Can we have a "release" build?

 

Sure.  I compiled it as Release with Debug Info, so if there were any issues, they could be fixed, since I don't think there are any other 64bit versions for windows like this.  But I can do a Release build without debug info if space for you is an issue.

 

I'll plop it on my source port build thread so I don't pollute Ketmar's one with too much of my babbling :)

[ketmar]

20 minutes ago, Gibbon said:

I'll plop it on my source port build thread so I don't pollute Ketmar's one with too much of my babbling :)

no problems, i'm ok with any k8vavoom talks here. ;-)

 

also, all, does this screenshot look interesting?

Spoiler

 

Spoiler

yeah, i'm implementing KVX/KV6 voxel model loader. nope, GZDoom VOXELDEF won't work, you will need to create the normal k8vavoom 3d model definition XML to use voxels. because from the engine POV voxels are just another type of 3d models.

 

[Mr.Rocket]

Sweet!

Is there a little documentation on the XML script or perhaps an example wad?

Thanks!

[ketmar]

3 hours ago, Mr.Rocket said:

Is there a little documentation on the XML script or perhaps an example wad?

sorry, no documentation yet. it is greatly extended original Vavoom model definition script, but the original didn't had any docs too. ;-) anyway, voxel converter is really dumb, the resulting armor model has more triangles than the whole e1m1, lol. it is more a kind of PoC than something really usable right now.

 

the converted model is cached, tho, so i will prolly write a better converter later. first time loading will be very slow (it's not fast even in its current form, that's why i added caching), but i hope to cull more triangles. basically, most flat surfaces can be joined into bigger quads, and textured with a proper bitmap texture instead of a solid color. this should greatly reduce triangle counts for ammo and health pickups, for example (because they are mostly boxes).

 

of course, it is playable even in its current form (models are uploaded to GPU only once), but i can do better! ;-)

 

for now, i manually converted Doom Voxel Project definitions for testing purposes. i may upload it with the next public build. actually, it is downloadable right now, but it won't work in public builds, of course.

[pramenid]

> also, all, does this screenshot look interesting?

 

It does!  I was holding off asking for this :) 

 

Looks good, but I might recommend converting from https://www.moddb.com/mods/reikalldoomvoxels, which seems to be more up-to-date.  And it covers many more environmental models (barrels, trees, lights, etc)... and has nicer healthpack models, IMO. 

 

Spoiler

 

[ketmar]

1 hour ago, pramenid said:

Looks good, but I might recommend converting from https://www.moddb.com/mods/reikalldoomvoxels, which seems to be more up-to-date.

thank you! i did Doom Voxel Project simply because i needed something to test the things with, and that was what i had at hand. the feature is very WIP yet.

[Steinkrauz]

On 3/11/2022 at 6:11 PM, Mr.Rocket said:

Is there a little documentation on the XML script or perhaps an example wad?

I wrote something based on the source code

https://github.com/steinkrauz/k8vavoom_extras/blob/main/K8_model_definition.md

 

@ketmar, please have a look, may be you can add some hints too or fix an error.

[ketmar]

11 hours ago, Steinkrauz said:

please have a look, may be you can add some hints too or fix an error.

heh. there is "specs/unfinished/k8vavoom_models.txt" file, that briefly describes things. it was created when i realized that i myself don't remember what tags and attrs i added to model definitions. ;-)

[ketmar]

so i cleaned up and rewrote some parts of model parsing and rendering. as a consequence, GZDoom MODELDEF support is way better now, and match GZDoom rendering in more cases.

 

one fun effect of treating voxels as yet another 3d model format is that i can use all Vavoom model definition power. for example, it is possible to attach several models to one sprite frame; each such "submodel" can have its own scale/offset/rotation, it is possible to set "fullbright" flag for any submodel, and any submodel can be made translucent. so it is possible to recreate Reikall "multivoxel" animations without any DECORATE. also, it is easy to create Reikall lamps with a translucent glass. here's the screenshot: each lamp consists of the lamp voxel, and a translucent "glass" voxel. it's all done only with k8vavoom model definition XML, no DECORATE required.

 

Spoiler

 

p.s.: yah, it means that k8vavoom voxel support is more advanced than in GZDoom. while technically GZDoom is able to do the same thing, there is no way to describe it with VOXELDEF. ;-) (and that's why there will be no VOXELDEF support; sorry, it is just too limited.)

Edited March 14, 2022 by ketmar

[ketmar]

so, i implemented Very Experimental Voxel Optimiser.

 

before (look at the FPS number):

Spoiler

 

after (look at the FPS number again):

Spoiler

 

the price for that is The Infamous Space Ants. i.e. voxel models can have occasional flickering white pixels. there is nothing i can do with it: adding more vertices to get rid of "space ants" will bring FPS down to unoptimised mode again. but optimisation level is selectable from the video options menu (requires engine restart), so you can trade quality for speed, as usual.

 

Spoiler

for those who are interested: normal conversion method (used by DelphiDoom, and i believe by GZDoom too) is to create one quad for each visible voxel side. this gives the perfect image quality, but even the simple object like armor has ~4800 triangles. in optimised model, the engine tries to merge horizontal or vertical quad runs, so the same armor model has only ~1400 (or ~1300 with even more optimisations) triangles. but this optimisation creates T-junctions, and hence Space Ants. adding vertices at each T-junction will bring vertex and triangle numbers almost back to the unoptimised numbers, so Space Ants will not go anywhere. they aren't that much noticeable, though.

 

Edited March 15, 2022 by ketmar

[jval]

1 hour ago, ketmar said:

so, i implemented Very Experimental Voxel Optimiser.

 

  Reveal hidden contents

for those who are interested: normal conversion method (used by DelphiDoom, and i believe by GZDoom too) is to create one quad for each visible voxel side. this gives the perfect image quality, but even the simple object like armor has ~4800 triangles. in optimised model, the engine tries to merge horizontal or vertical quad runs, so the same armor model has only ~1400 (or ~1300 with even more optimisations) triangles. but this optimisation creates T-junctions, and hence Space Ants. adding vertices at each T-junction will bring vertex and triangle numbers almost back to the unoptimised numbers, so Space Ants will not go anywhere. they aren't that much noticeable, though.

 

 

Good to see progress in voxel rendering!

 

In DelphiDoom using one quad for each visible voxel side is the worst case scenario, i.e. there is not even one voxel element with a neighbor with the same color in the same plane (x/y/z).

[ENEMY!!!]

I don't know if this is already known but at present there are compatibility issues with ANIMDEFS.txt in the pk3 structure for texture resources.  The animated textures in OTEX_1.1.pk3 don't work (but the flats work fine), but in OTEX_1.1.WAD everything works. 

 

I'm aware that development is inevitably slowed by the war, and for now a workaround which I've tested successfully is to apply TEXTURE2/PNAMES or TEXTURES lumps within the root of the pk3 file (e.g. copying TEXTURE2/PNAMES across from OTEX_1.1.WAD to OTEX_1.1.pk3).

[ketmar]

32 minutes ago, jval said:

In DelphiDoom using one quad for each visible voxel side is the worst case scenario, i.e. there is not even one voxel element with a neighbor with the same color in the same plane (x/y/z).

i can convert the whole horizontal or vertical run of voxels as one quad, regardless of their color. that's what optimisation levels higher than "none" do. of course, it inevitably produces T-junctions…

 

also, those FPS numbers are for shadowmapped renderer. lightmapped renderer on the same scene easily hits 100+ FPS. shadows are costly! ;-)

 

5 minutes ago, ENEMY!!! said:

I don't know if this is already known but at present there are compatibility issues with ANIMDEFS.txt in the pk3 structure for texture resources.

it is using "range" animation, which is highly dependent of file order in an archive. TEXTUREx lump overrides the order, but without it, the engine is confused. the whole animation system in Doom is… well… let's say, it's not the best one. tbh, i am simply afraid of touching animatin sequence building code, it is a freakin' mess, and prolly nobody knows how it works now. ;-)

[Steinkrauz]

On 3/15/2022 at 9:13 PM, ketmar said:

implemented Very Experimental Voxel Optimise

 

https://github.com/steinkrauz/k8vavoom_extras/releases/download/220315/K8Vavoom220316.zip

 

For you, my fellow early adopters ;-)

Edited March 16, 2022 by Steinkrauz : link update

[pramenid]

Nice!  Looks good using the k8-voxels-afznc7.pk3 pack from last week.  I am curious, you showed examples reikall voxels in the screenshots, but the latest build doesn't seem to recognize those .pk3s directly (yet).  So is there another version of k8-compatible voxel assets based on that?

[ketmar]

32 minutes ago, pramenid said:

So is there another version of k8-compatible voxel assets based on that?

yes, i converted most of them too. GZDoom VOXELDEF is not supported, and will never be (due to technical incompatibilities, not because i haet GZDoom or something ;-), so GZDoom voxelpacks must be manually converted. and i'm not sure if i can distribute k8vavoom reikall pack, it's not clear from the readme if it is counted as "another work", or not.

[ketmar]

and i implemented slightly better voxel optimiser, and got +2 FPS on that test scene. i still have some optimisation ideas, but it will require at least x3 more code for another 2-3 FPS. maybe i'll do it in some distant future…

 

just for fun, the same scene with lightmapped renderer:

Spoiler

 

[pramenid]

I am curious -- it seems you are rendering voxels as geometry.  Is it feasible to render them using GL_TEXTURE_3D, using GL_NEAREST filtering?  Then the geometry is a box.  Do the models use the Doom 256-color mapping and a transparent color index?  It seems then that models could trade off memory cost (some 64x64x64 model being 256k) but with very simple geometry.

 

n.b. I have never done this before and may be missing some obvious issue with this approach... but thought I'd get it out of my head before I went to sleep.  ;)

[ketmar]

7 hours ago, pramenid said:

Is it feasible to render them using GL_TEXTURE_3D, using GL_NEAREST filtering?

it can be done, but it has little sense. you'll waste a lot of memory (effectively trashing GPU memory caches), and you will still need a custom raycasting GLSL shader (effectively trashing GPU cores with missed branches). actually, a set of slightly different shaders, to support dynamic lighting.

 

you technically can render voxels with GPU help quite fast, but you will need an engine designed and written exactly for this kind of geometry, and prolly OpenGL4+, for tesselation and compute shaders.

 

just note that programming for GPU is very different from CPU programming. you have a different set of tradeoffs and things to avoid, and you'd better remember about GPU texture and vertex caches, and… ah, it's a whole new ball game. ;-)

 

p.s.: also, KVX files have their own palette (yet it is usually Doom one), and KV6 files are using RGB colors.

Edited March 16, 2022 by ketmar

[ketmar]

fun fact: most KVX objects have some invisible voxel faces marked as visible. Boss Cube from reikall, for example, is empty inside, but its contour voxels marked as "visible from inside", and the engine has to create triangles for the faces the player could never see anyway. i implemented visible face recalculation algorithm, and the result is:

 

before: 11065 unique vertices, 5556 triangles

after: 9587 unique vertices, 4820 triangles

 

it may not look like a big improvement, but using this algo i managed to get almost +2 FPS on the test scene.

Edited March 16, 2022 by ketmar

[pramenid]

Cool, thanks for the explanation.  I knew something was off about my bright idea ;) 

[ketmar]

so i thought a little, and decided to implement "space ants eliminator" too. it may bring a lot of triangles back (around 1/3 of the whole triangle count by a rough estimation), but hey, if you have a fast GPU, or a small number of voxel objects, or just want to trade some speed for some quality… you should be able to do so. i already wrote brute-force implementation, but it is O(n^2), i have to optimise it before commiting. ;-)

[ketmar]

good optimisations can make wonders! t-junction fixing time on quite a big voxel model decreased from 28 seconds to 31 milliseconds. it is basically O(n) now (where `n` is the total number of edges).

 

it is also now possible to change voxel optimisation mode without restarting the engine.

[ketmar]

On 3/15/2022 at 10:15 PM, ENEMY!!! said:

I don't know if this is already known but at present there are compatibility issues with ANIMDEFS.txt in the pk3 structure for texture resources

btw, thank you, i created a ticket for this. so don't worry, the report won't go to /dev/null, i will eventually take a look. ;-)

[ketmar]

i simply can't stop. tried Yet Another Voxel Conversion Algo, which is slightly slower on startup (still almost instant for DVP voxels, tho ;-), and managed to make triangle count even smaller: it can cut from 100 up to 1000 triangles sometimes.

if you're interested, even the simpliest k8vavoom converter is much better than the one in GZDoom. GZDoom is able to merge only voxels in vertical runs ("slabs") that have the same color. even the most simple k8vavoom converter creates one quad for any number of slab voxels regardless of color. more advanced converter can create such runs in 4 directions, not only vertical. and the new algo is able to merge voxels on any plane into even bigger quads. i believe that this is the best (almost) real-time voxel -> 3d mesh converter among all existing "retro" 3d engines. because i am smart as… ahem, nevermind.

 

p.s.: the new algo is not in the repo yet, i'm still polishing it. also, porting my conversion code to GZDoom won't be hard: it is mostly self-contained library that doesn't depend on the rest of the engine, and can produce texture, vertices and indicies to render converted models as triangle soup, or triangle fans. of course, any other sourceport can take it too. just wait until i finish and polish it. ;-)

Edited March 19, 2022 by ketmar

[ketmar]

just for fun: standalone converter (my testbed), in "ultra" mode, with t-junction fixer.

 

the famous Voxlap test scene, fully converted:

Spoiler

 

some stats:

Spoiler

loading: _vox00/untitled.vxl (with normals)
xsiz=1024; ysiz=1024; zsiz=256
loaded in 938ms; grid size: 1024x1024x256
fixed 1825756 voxels
optimised in 32.244
building slabs...
*** created compressed voxel data; 33584853 bytes for 1024x1024x256
ATLAS: resized from 8x8 to 16x16
ATLAS: resized from 16x16 to 16x32
ATLAS: resized from 16x32 to 32x32
ATLAS: resized from 32x32 to 256x64
ATLAS: resized from 256x64 to 256x256
ATLAS: resized from 256x256 to 256x512
ATLAS: resized from 256x512 to 512x512
ATLAS: resized from 512x512 to 1024x1024
ATLAS: resized from 1024x1024 to 1024x2048
ATLAS: resized from 1024x2048 to 2048x2048
*** ultra conversion done in 1:54.856
basic conversion: 498462 quads (996924 tris).
converted in 2:00.767; optlevel is 4/4
color texture size: 2048x2048
498462 quads, 996924 tris, 1993848 unique vertices (of 1993848)
OpenGL data created in 1.021
334031 unit quads found.
1993848 edges found (996924 tris, 1993848 verts)...
vox dims: (-733,-610,-141)-(290,413,109)
grid size: (1024,1024,251)
180130 t-fix vertices added (180130 unique).
rebuilt model: 1188394 tris, 2179648 vertices.
t-junctions fixed in 420ms
fixed tris: 1188394

 

[Lobo]

Impressive!

[ketmar]

and the new "voxelib" is in the repository now. you can simply copy two voxelib files (header and cpp) to your project, and uncomment `VOXLIB_STANDALONE` in the header. the library is usable in any C++ project, and doesn't require anything except the standard C99 headers (yes, not even STL!). it includes loaders for KVX and KV6 formats, so you don't even have to write them yourself!

 

so if you need to convert voxel model to 3D mesh, don't look any further, "voxelib" is here! ;-)

 

p.s.: WARNING! beta quality! ;-)

[Steinkrauz]

Just another voxel project to borrow models from. I've added some gore models to my vavoom and it looks quite nice.

 

EDIT: removed the offending link

Edited March 29, 2022 by Steinkrauz