k8vavoom: no good thing ever dies!

[ketmar]

i never even dreamt to get the mapper of such caliber to make maps for k8vavoom. i only hoped for some usual quick "demo" maps from someone, not something that looks like this, and was actually made to be fun to play. and if somebody thinks that i'm praising it all too much... nope. you all can't even imagine how excited i am. (insert Droopy "i am happy" gif here)

[Mr.Rocket]

I can't agree more, k8vavoom is an awesome port and I can't wait to see moar!

I went to the trouble of fixing up a launcher for it, although not done.. currently working on a map at the moment..

Will finish it soon.. and I and love what is already popping out of the woodwork!@Remilia Scarlet ;)

You guys rock! ;)

 

I did a little work on the D3Retro project, which a lot of it I did had to do with the brightmaps. I setup some sample maps as well so it would be somewhat easy to see what was what. It turned out ok, and I think we were on that thing for damn near a year. 

I doubt that any map I make could hardly be as beneficial as Scarlet's, while I'm an artist of sorts (i like to venture into effects ~ in different games etc), Although don't be mistaken.. Scarlet IS the true artist here, and I can't wait to see it. ;)

 

Edited March 8, 2021 by Mr.Rocket

[ketmar]

thank you for your kind words, @MidnightMage and @Mr.Rocket!

 

and just in case -- i never wanted to say that some mappers are worser, or not needed anymore (lol), or something. even the best engine in the world won't make a game by itself, and won't play itself. thank you all, people -- mappers, players, musicians, gfx artists, and all others. let's kill some demons and save the world... again! ;-)

 

 

[furyweb]

How's the model support for k8vavoom. I am thinking of a stand alone game/ tc using this source port.

[ketmar]

1 hour ago, furyweb said:

How's the model support for k8vavoom.

MD2, MD3. it is better to use native vavoom xml model definitions, tho, gzdoom MODELDEF emulation is incomplete.

[3t0]

Hmm, so I built new version and it seems that under stencil + shadowmap mode I am really not seeing any sparklies so far, congratulations!

 

Do I understand it right that lightmapped renderer is completely separate codepath that shares only very small amount with stencil+sm one? Or how come that only in stencil+sm path t-junctions get fixed?

 

For now I always played on lightmaps, but I am surprised with performance of stencil+sm mode, at least in gunrock's (I think it was them) remasters I am getting constant 130 fps no sweat, same for vanilla IWADS.  Of course my daily drawer is nothing much, compared to shit avail these days, but still:

xx:xx.x VGA compatible controller: NVIDIA Corporation GK106GLM [Quadro K2100M] (rev a1) (prog-if 00 [VGA controller])

Unfortunately while Remilia's map is indeed beautiful (it took me some stalking and digging around to get to the map, as I am bilnd :)), and good job by the way, the "overloaded" areas there fry my measly rig at 25-32 fps. This was a bit disappointing for me to see.

 

Sadly I am really overloaded with work here, so I haven't had much time to observe doom-world happenings.

 

But still, the little amount of time I have, and your (ketmar@) analysis of doom movement code (eg. "slapping on of 3rd dimension onto what is essentially 2d movement base code" and resulting inconsistencies from that) and recent release of "final" Arcane Dimensions, made me once again, investigate Quake's worlds more.

 

What really surprised me is current frame rate "stability" of all major Quake descendants, especially that of Quakespasm and Quakespasm Spiked but also of FTEQW as well. On the other side, all the (major) modern doom engines, including GZDoom and k8Vavoom (I haven't tried KEX stuff like Doom64 (I guess is the game) which I suppose is quite different internally), have performance problems, when you enable too much shit. Don't get me wrong, vanilla levels almost always play rather well and solidly, but at certain point, no matter which custom modern wad and engine combination I pick, my machine starts to deep fry potatoes.

 

This made me wonder, is there something else (besides movement, which makes it essentially impossible to have solid physics in Doom), inherent in Doom descendants, that makes it impossible for modern Doom renderers to maintain Quake-like framerates and fuildity? Or is that really just cost of dynamic lighning models and codepaths that modern Doom engines tend to pick?

 

After being spoiled by Quake's constant framerate stability for last few weeks (during afterwork play), even in monster heavy mapjams (it seems like, that with current quakespasm/hardware generation (eg. hw in range -7 years - yes don't laugh), Quake seems now able to reach and maintain almost Doom level monster counts, while still keeping impeccable gameplay fluidity), playing some new doom maps was a quite jarring feeling. This was little disheartening to experience, as I was always big Doom fan.

 

I do realize this is a bit unfair to Doom engines (apples to oranges), after all I guess, Quake is really much simpler internally now, than any modern Doom engine is, making heavy use of baked-in lightning, multi-texturing and full scene single-pass rendering, it can probably spit frames much faster than any Doom with PBR and stuff, but I would still expect at least k8Vavoom in lightmap mode to be able to outperform, or at least to keep up with it. This does not seem to be case, so I am curious whether there is an opinion about possible causes of this?

 

After all, Doom weaponry and bestiary still seems to be the best one around, even when compared to Quake. After some time you get tired of nerfy shotguns and spongy Quake monsters higher up the ladder and their embedded unfairnes (Pinkies are really cute little puppies when compared to Fiends, Shamblers are equivalent to Archvile evil (yes I hate Archviles) and they don't even revive the dead, and Vores and that damned Slime things are fucking ridiculous). I would really welcome something that plays like Doom but maintains Quake's stability. So is this even theoretically possible with current Dooms (and where they are heading)?

 

Finally, how much of k8Vavoom VavoomC vm is directly from QuakeC vm?

[ketmar]

1 hour ago, 3t0 said:

Hmm, so I built new version and it seems that under stencil + shadowmap mode I am really not seeing any sparklies so far, congratulations!

thank you!

 

1 hour ago, 3t0 said:

Do I understand it right that lightmapped renderer is completely separate codepath that shares only very small amount with stencil+sm one?

yes. basically, the renderer consists of two parts: high-level, which does BSP walking and collects everything that need to be rendered, and the actual low-level rendering code, that feeds the GPU. high-level code is the same for all renderers, but low-level is completely different. lightmapped renderer also does alot of subdivisions of generated surfaces, for better lightmap management. so basically, lightmapped renderer has more touching polys, and it is alot harder to fix t-junctions in it. so i decided to implement it at least for other two (actually, for another one, because stencils and shadowmaps sharing 99% of code) for now.

 

1 hour ago, 3t0 said:

This made me wonder, is there something else (besides movement, which makes it essentially impossible to have solid physics in Doom), inherent in Doom descendants, that makes it impossible for modern Doom renderers to maintain Quake-like framerates and fuildity?

the movement code has nothing to do with it. the secret is that Quake (and all modern 3D engines) has immutable geometry. in Quake, if something's moving, it is a completely separate 3D model (immutable too). all doors, lifts, etc. are separate models (brushes, in Quake terminology). yet in Doom, any geometry may change at any time. yes, vertices are moving mostly along one axis, yet this is enough to completely kill all fancy optimisaions possible on modern GPUs. with Quake, i can upload all level data to GPU, create draw lists, and then simply render BSP leaves with one OpenGL call, transferring only several bytes. but with Doom, i not only have to transfer level data in full each time, but i also have to check if it was changed from the previous frame, and recreate all changed surfaces.

 

i.e. with Quake, i can precaluclate almost everything, most of that data can be uploaded to GPU once and used without changes, and i don't have to do any checks and rebuilds. with Doom, it is completely opposite. and writing sophisticated analyzers to determine which part of a Doom map is immutable simply doesn't worth the efforts: they will break on any more-or-less complex map, or on any map with scripts. i.e. exactly on the maps that needs to be accelerated.

 

of course, lighting adds some overhead too, but the core issue is what i described above.

 

also, Quake maps are simplier. it may sound crazy, but modern Doom maps are very complex. they have alot of linedefs for detailing, and the engine have to resend alot of data on each frame. and most (all?) Doom engines are using 1D clippers to cull invisible geometry -- which basically means that any window is basically "transparent". even if you cannot see some window because it is higher than the window near the camera, this "window through window" situation completely prevents geometry culling. so the engine may render alot of map you cannot possibly see (and it has to send all that data to GPU on each frame).

 

so the choice is simple: either support only vanilla-style maps (id did it in their new Doom Eternal) and be fast, or support everything Doom mappers are throwing at us, and be slow.

 

1 hour ago, 3t0 said:

So is this even theoretically possible with current Dooms (and where they are heading)?

i don't think so. this could be possible only if we'll throw away most of the maps created so far, introduce new mapping standards, rewrote all editors, and somehow force people to switch. totally unrealistic scenario. ;-)

 

1 hour ago, 3t0 said:

Finally, how much of k8Vavoom VavoomC vm is directly from QuakeC vm?

zero. Vavoom is more inspired by Unreal, and it has stack-based VM, as opposed to Quake register-based. this is something i'd like to change, but Janis left us without compiler/VM testing suite at all (literally, zero tests), so i won't take that risk. at least for now.

 

 

p.s.: Vavoom does GPU caching for map 3d models (those that replaces sprites), tho. so you can have thousands of 3d pickups and decorations without much trouble (and it can even be slightly faster than sprites). the power of immutable geometry!

Edited March 9, 2021 by ketmar

[Remilia Scarlet]

59 minutes ago, 3t0 said:

Unfortunately while Remilia's map is indeed beautiful (it took me some stalking and digging around to get to the map, as I am bilnd :)), and good job by the way, the "overloaded" areas there fry my measly rig at 25-32 fps. This was a bit disappointing for me to see.

For what it's worth, this is about how well it runs for me in the most intense areas.  Other areas are perfectly fine and hit 200+ fps.  This map is almost three years old, and I've learned a lot about more efficient uses of map geometry and lights since then (mostly from Quake mapping, as it happens).  Nothing I've made since then slows down this much in k8vavoom, so I'm guessing it's just a sub-optimal map.  I could have modified it to reduce unneeded geometry and lighting, but as I hinted at in the release thread, I also would have to do a LOT of destructive editing to get it equal to how I'd build it today.  And that's something that I simply didn't want to do.

[ketmar]

also, don't forget that shadowmapping code is still very young. i may find some way to make it faster in the future, but for now it has almost no optimisaions (besides heavily optimised shaders; please, don't try to decipher them if you value your sanity even a little). maybe i'll find some way to partially cache lighting data, or something. or maybe not. ;-)

[Mr.Rocket]

I thought it might be interesting if when the player is invisible and of course doesn't cast a shadow, it might be a neat feature to have the invisible player or Things/sprites cast maybe a dim yellow or amber color shadow as an option.

It would be kind of freaky to see only a slight shadow enter a room when another player etc. is invisible. :P 

This of course as an option and likely disabled by default.

Edited March 10, 2021 by Mr.Rocket

[ketmar]

43 minutes ago, Mr.Rocket said:

t might be a neat feature to have the invisible player or Things/sprites cast maybe a dim yellow or amber color shadow as an option.

sadly, it's quite hard to implement. basically, shadow is not modulating the light, it either completely blocks light, or not. modulating will be much slower. also, translucent objects should not block other shadow casters, and this will slow down the whole shadowmapping by 2-5 times at min (it will require more texture storage, and shadowmap texture should be completely cleared and re-rendered for *each* light source; it is already quite slow -- that's why default shadowmap size is 64x64, for example).

 

the idea is neat, yeah. but technical limitations... alas. so translucent objects simply cast no shadows at all, and i don't think that i'll find any way to change this, and still keep the engine realtime. ;-)

[3t0]

Sorry I could not help myself, so this one of those longer ones but please, consider reading it.

 

18 hours ago, ketmar said:

lightmapped renderer also does alot of subdivisions of generated surfaces, for better lightmap management. so basically, lightmapped renderer has more touching polys, and it is alot harder to fix t-junctions in it.

I did not realize how much lightmapping affects the whole thing (I see now, in quakes everything is pre-subdivided by the map compiler, but k8vavoom must do it online) - I guess this this one is not your favorite part of the code.

 

17 hours ago, ketmar said:

the movement code has nothing to do with it.

I understand that movement code has nothing to do with rendering performance. What I was trying to say is that, from your analysis too, numerical stability of Doom's movement code is just very lucky coincidence, or rather: it woks relatively solidly for what it is, but result is basically achieved by an very ad-hoc solution. As I understand it now, no 3d matrices, or quaternion multiplication ,or any other unified standardized approach is used, instead, if I understood it right, for top down 2D movement and rotation vector addition and complex numbers multiplication are used(?), and then 3D z-element is handled ad-hoc on case by case basis.

 

This means that doom movement has it's own peculiarities that are doom movement relevant only. Solving this issue by ripping out the code and replacing it with more proper 3D movement code, like in quakes or other 3D games, is of course possible, but it would then irrevocably and terminally, broke the game for purists (I imagine SR50 thing would now not work, so goodbye speedruns and so on), or basically, everybody else. Players have spent too much time in honing their skills to the current system, so it's essentially impossible to replace it, as it would then be a completely different game. As I understand it, this makes it impossible for this "issue" to ever be "really fixed". This movement is still very good solution for Doom and similar games, giving them "a doom-y feeling", but it basically became a design constraint.

 

It took some time for this to sink in (like 20 years :) ) and it was only your explanation of the algorithms, that was sufficiently clear enough, about the state of how things are done, for me to finally grasp this. This also made me realize why other engine coders are so reluctant to touch the movement subsystem - it's such a sensitive subsytem it would immediately blow up with players and other coders into a lot of drama, breaking the game and mods in the process (I already read somewhere how zdoom movement code is not sufficiently vanilla). It would be cool feature to have for "new indie" game, but besides Hedon (which I am great fan of) and that "rabbit thing", and perhaps few platformers in the making, we haven't really seen much of standalone games with doom engines. Optimizing things for that use case would be like ignoring 99.9999% of actual engine users. So I finally realized, that unless one is willing to maintain their own full fork (there are few such corpses on github cemetery) it's essentially impossible to change the movement code in any major way (unless some genius of course comes up with a way how to do it somehow, which I doubt).

 

18 hours ago, ketmar said:

The secret is that Quake (and all modern 3D engines) has immutable geometry. in Quake, if something's moving, it is a completely separate 3D model (immutable too). all doors, lifts, etc. are separate models (brushes, in Quake terminology).

Yes I was aware of that, I still have not realized the implications fully. So the immutable geometry is basically only way to go with modern hardware - you upload all your stuff into the card, including the buffers and shaders and then let it handle shit from there, with only "few" handful PCIe control commands passed back and forth. This way CPU is free to do other shit and GPU can do it's thing too, memory bandwidth is not eaten, as they both are more or less separate entities. No costly geometry uploads killing the transfer bus. Got it.

 

18 hours ago, ketmar said:

in Doom, any geometry may change at any time. yes, vertices are moving mostly along one axis, yet this is enough to completely kill all fancy optimisaions possible on modern GPUs.

Hmm, so even if vertices are bound to their position by BSP rigidity and unable to move on X,Y (topdown) plane, the Doom geometry is still not really immutable, because Z-height can change at any time (lifts, doors - hmm, this is neat hack though, it allows to "reuse" BSP tree for any height :) ) - and yes we are ignoring polyobjects for now, which can move on XY axes, but don't have Z dimension - talk about limitations!).

 

18 hours ago, ketmar said:

and writing sophisticated analyzers to determine which part of a Doom map is immutable simply doesn't worth the efforts: they will break on any more-or-less complex map, or on any map with scripts. i.e. exactly on the maps that needs to be accelerated.

Hmm at first, I thought you just need to analyze sector-linedef "action links" and you are all set, but then I realized with ACS (and other scripting) you can change any sector height at any time right?

 

18 hours ago, ketmar said:

so the choice is simple: either support only vanilla-style maps (id did it in their new Doom Eternal) and be fast, or support everything Doom mappers are throwing at us, and be slow.

So they might be doing exactly "the action links" analysis, I naively thought up: they can pick out immutable clusters of level (everyhting that is not a lift or door), converting those into immutable geometry buffers, and gain rendering speed, but then we would lose the mutability, that advanced level designers are actually after. Got it. So basically due to freely movable sector heights, we are unable to get speed optimizations, despite the fact vertices are rigidly pinned down in X,Y plane by BSP ... that is so sad.

 

18 hours ago, ketmar said:

Doom engines are using 1D clippers to cull invisible geometry -- which basically means that any window is basically "transparent". even if you cannot see some window because it is higher than the window near the camera, this "window through window" situation completely prevents geometry culling. so the engine may render alot of map you cannot possibly see (and it has to send all that data to GPU on each frame). 

I always wondered about this - could not height calculations be somehow factored in? Something like Z-respecting 2D view cone/triangle clipping in secondary pass? Or it's not worth the effort?

 

But anyway, in current state of affairs, this basically means, that all doom engines will always be stuck in something akin to semi-immediate OpenGL mode shuffling most of the geometry stuff back and forth between the CPU and GPU with expensive change detection passes in between, all this to accommodate for "possibly anytime" z-moving geometry. Hmmm, all this is strongly logical, but rather sad ...

 

... but I am the guy who tends to ask stupid questions and comes up with stupid ideas, so please, let me to stir the hornet nest a tiny bit more.

 

When I read about your 1D clipper treatise, I realized, that it always struck me strange, why engine coders are reluctant to offload these things onto the ultimate accelerator eg.: mappers themselves.

 

This IS old school thinking (nowadays younglings are brainwashed into mantra: "throw more hardware software and ai at the problem as human time is much more expensive") but map designers themselves are the "cheapest" optimizers available (just like experienced assembly coders are).

 

More over, by now, it is pretty clear, that doom exists outside of time (like this CPU-GPU mismatch proves), and people willing to map for it must "degrade" themselves into "back then " mindset anyway, whether they want it or not. Mapping for doom is such an act of labor and love, that mappers learn and "waste" their time with elaborate doom-only techniques anyway. They also become doom experts in the process as well. So they are gonna invest their time anyway.

 

Finally, there is this company which abused its map designers often, at the expense of the cpu time, and it turned out to work for them pretty well: Valve.

 

Anybody who tried mapping for Half-Life back then, pretty soon learned that without navigational "node mesh", the "phenomenal" half-life AI is pretty dumb (I dare to say stuck at Duke3D level, which is abysmall). AI info nodes are its meat and potatoes, and without them, HL AI is almost clueless. So, if you wanted to have good play for your SP map (as a level designer) you had to place those damn nodes to annotate the environment for AI to use. These days, there are packages, that can do this semi automatically (for other engines), but back then, it was serious labor. Once you had everything meshed up, if you decided to change a level now and your node meshwork became outdated, good luck: you had to rewire everything again. So this was Valve's solution, not something that Carmack or Romero would do (who liked to minimize and automate such extra work as much as possible).

 

What surprised me even more, was introduction of occluders in Half-Life 2. Because it's still borked Quake 1 engine with some slight polish :), there seem to have been some problems with geometry performance, similar to what we are discussing here. Instead of solving this programmatically as Carmack would do, Valve introduced a new brush function type instead, to occlude geometry based on toggle-able state and possibly distance (if I remember correctly) and offloaded the optimization work to the mapper. Now whole areas of the level could be turned off or on visually, with flick of a trigger. If I remember correctly, this has been used to toggle immutable part of geometry (like streets behind windows and such).

So where am I going with this whole thing? Would a new linedef that could act similarly as this HL occluder (toggleable by switches/ACS and what have you), eg turning visibility of whole chunks of level based on player position/triggers, help out?

 

I went trough several line action specials but I have not seen any Line_BlockVis or something like that there. Of course this poses a problem when line visblocking state desyncs with player state (but there are other things in modern dooms already that can desync too), but this would be in full control of the mapperm so it ould be their responsibility in keeping this working. I think it would be pretty "small" modification, with a possibly huge gain, that could help advanced mappers a lot. Or no? Would it be doable? And if yes would it be worth it?

 

While accepting this "offloading onto a mapper" mindset and thinking a lot about this immutability thing since yesterday, yes I could not sleep well, second idea came to me. Why not then just offload immutability calculation onto designers as well?

 

Let me elaborate: with vanilla levels, immutability is a given and mutability detection is simple: everything that has action link (door, lift) is mutable, everything else isn't, period (there are probably few special cases, but this is manageable as you said.

 

Now what is problematic is mutability of advanced levels, the ones, that would benefit from immutable acceleration the most, as you wrote. But there might be a way: old formats proved to be insufficient for the advanced mapping anyway, so I would dare to say, that majority of adavnced mappers is now using UDMF for everything. UDMF has many interesting properties (like incredible zip compressablity - as it is pure text) but the thing that fascinates me the most about it, is its insane extensibility. It is basically relatively simple key/value storage system, with few extra data structures tucked, that can accommodate addition of new features very easily.

 

This means that it would be possible to define new subclass of UDMF map, one with "DEFAULT_IMUTABLE" feature flag set, where all sectors are immutable by default, but other sectors still have mutability possibility, through, for example, per sector settable mutability hint/flag.

 

This would map well to mapper workflows, and would require only slight editor extension (it's basically just addition of new prop per level and checkbox per sector in sector dialog). Usually, in a map, ~90% - ~99% sectors are immutable anyway, so for normal geometry, there would be no additional workload. Now once you have lift or door or any Z-movable sector for that matter, this one would not be valid/fully usable, until marked for mutability as well. Now, if you tried to Z-move immutable sector, and by any means (action special, ACS, Zscript, whatever), eg one that is not marked for mutability, engine would refuse to do so, spitting up error: "Context x{action special, ACS,....} tried to z-move immutable sector: xxx". This way both mapper and player would be notified immediately, that there is error in the map, at runtime, which is important as we both agree. If sector was marked as mutable however, it would move up and down just fine as intended.

 

Naturally, this would impose more work on mappers, as each time they make a door or a lift or any scripted area, besides creating action links and scripts, they would have to explicitly mark affected sectors for mutability as well.  But it would also provide great benefit to engines themselves. The engines could now churn through all those immutable "static sectors" and compile them into efficient display lists for very fast rendering. Mutable sectors would, of course, have to be handled same way as they are currently, but with this approach they would automatically became special cases, instead of being default case.

 

However now, no complex and error-prone mutability analysis would need to be done at engine level, as all that work would be offloaded to mapper, who is single final and ultimate instance, that knows which part of map needs to be mutable and which not, anyway. While this would introduce some extra workload on advanced mappers, it could also, potentially, offer great gains in performance, while still keeping most of the things as they are, for compatibility with older maps and whatnot. I don't know how much it would complicate engines internals, as there would now be two codepaths - one very fast for immutable level geometry and then the other, slow one for everything else, that is movable. But who knows, maybe it would still help internal engine organization.

 

I realize, this would be pretty big change community wise, requiring coordination and getting many other individuals and teams on board, but what I am interested in for now is technical feasibility and implementability and implementation cost, than marketing and actual realization.

 

Do you think it would be worth it, given the work and changes required?

[3t0]

20 hours ago, Remilia Scarlet said:

For what it's worth, this is about how well it runs for me in the most intense areas.  Other areas are perfectly fine and hit 200+ fps.  This map is almost three years old, and I've learned a lot about more efficient uses of map geometry and lights since then (mostly from Quake mapping, as it happens).  Nothing I've made since then slows down this much in k8vavoom, so I'm guessing it's just a sub-optimal map.  I could have modified it to reduce unneeded geometry and lighting, but as I hinted at in the release thread, I also would have to do a LOT of destructive editing to get it equal to how I'd build it today.  And that's something that I simply didn't want to do.

No problem, I hope for more modern k8vavoom maps form you instead.

[ketmar]

@3t0 i think we'd better use spoilers for our conversation, to not scare people away. ;-)

 

Spoiler

 

3 hours ago, 3t0 said:

I did not realize how much lightmapping affects the whole thing (I see now, in quakes everything is pre-subdivided by the map compiler, but k8vavoom must do it online) - I guess this this one is not your favorite part of the code.

yeah. tbh, i simply afraid of touching it. i had to, but it still scares me.

 

3 hours ago, 3t0 said:

As I understand it now, no 3d matrices, or quaternion multiplication ,or any other unified standardized approach is used, instead, if I understood it right, for top down 2D movement and rotation vector addition and complex numbers multiplication are used(?), and then 3D z-element is handled ad-hoc on case by case basis.

yeah. it is even dumber than this: Doom simply does "microteleports" along the path, checking if object bounding box is touching any wall (this is quite fast, thanks to blockmap -- no need to traverse BSP tree, just check if any line in a grid cell intersects a rectangle; it's quite cheap). when "microteleporting" failed, it simply stops -- because monsters and projectiles usually cannot slide along walls.

 

for sliding, it simply using the last "good microteleport position", and then does... something. it is basically ray-line intersection check, but with some hacks.

 

height check is done in "microteleport" step: it simply calculates the new height (max floor height for all touched sectors), and then checks if height difference is less than "max step height".

 

3 hours ago, 3t0 said:

Solving this issue by ripping out the code and replacing it with more proper 3D movement code, like in quakes or other 3D games, is of course possible, but it would then irrevocably and terminally, broke the game for purists

yeah, basically this. i tried to replace it with proper box tracing (Q3 movement code), and it almost worked, but i dropped the idea. besides physics changes, the other thing is that "microteleport checks" are used not only in movement code. i have to keep "microteleport" part anyway, and if i cannot rip it out, there is no reason to have *two* code pathes for basically the same thing. this will only give us more bugs. ;-)

 

i still hadn't lost the hope to replace the movement code, tho. my first try was unsuccessful, but i may try some other ideas next time.

 

3 hours ago, 3t0 said:

So the immutable geometry is basically only way to go with modern hardware - you upload all your stuff into the card, including the buffers and shaders and then let it handle shit from there, with only "few" handful PCIe control commands passed back and forth.

not only this. with immutable geometry, alot of other things can be precaluclated too: PVS, grouping geometry by textures, and other things that can be used to optimise everything. for example, even if i'll upload the whole map to GPU, i will still need to group the geometry to minimise texture and shader switches (they're VERY costly too). so i still have to check polys on the CPU to create proper rendering commands.

 

4 hours ago, 3t0 said:

Hmm at first, I thought you just need to analyze sector-linedef "action links" and you are all set, but then I realized with ACS (and other scripting) you can change any sector height at any time right?

yeah. also, there are more complex actions, like stair builders, for example, and they can affect many sectors. besides that, most commands are doing things like "rise this sector's floor to the lowest floor height of all adjacent sectors". so you cannot even reliably calculate min/max limist, because adjacent sectors can be modified too with some of those commands, and so on... and simply creating texture atlases won't work, because Doom textures are repeating. we can emulate texture repeating with shaders, of course, but this is only one among many-many other "GPU-hostile" things in the engine...

 

4 hours ago, 3t0 said:

So they might be doing exactly "the action links" analysis, I naively thought up: they can pick out immutable clusters of level (everyhting that is not a lift or door), converting those into immutable geometry buffers, and gain rendering speed, but then we would lose the mutability, that advanced level designers are actually after.

yes, i'm pretty sure that this is what they're doing. there is simply no other way to efficiently do it in modern engines. i'm not even sure that they added support for all possible Doom actions (not all of them are used in vanilla maps, afair).

 

4 hours ago, 3t0 said:

I always wondered about this - could not height calculations be somehow factored in? Something like Z-respecting 2D view cone/triangle clipping in secondary pass? Or it's not worth the effort?

i'm still looking for a good way to do it. improving clipping code is something i'm constantly experimenting with, and i have several ideas to try.

 

the other problem here is that in Vanilla you cannot look up or down. even with Hexen "shearing" the camera is still doesn't change pitch. it is much easier to write an "exact" screen-space clipper for this case. but when you have arbitrary 2d polys (as a result of proper 3d projecting for non-zero pitch), everything becomes much more complicated here.

 

first Unreal, for example, solved this by always using software rasterizer (without texture mapping, of course) to maintain the list of "free" pixels, and checks polys against that list.

 

4 hours ago, 3t0 said:

When I read about your 1D clipper treatise, I realized, that it always struck me strange, why engine coders are reluctant to offload these things onto the ultimate accelerator eg.: mappers themselves.

eh... this idea is great, but it will never ever become the reality, i'm afraid. to make it real, we need all advanced sourceports to agree on that feature set, and to implement it. otherwise it will be hard to convince mappers that they have to run extra ten miles only to have their maps work slightly faster in sourceport SuperMegaDoomEngineMark666. i.e. at least GZDoom, Zandronum, Odamex, Eternity, DelphiDoom, k8vavoom should support exactly the same feature set. and Crispy, Retro, PrBoom should at least be aware of it. and it should be somehow shoehorned into original binary map formats, because... because just go and search for some topics on UDMF mapping on this forum, and you'll see why. ;-)

 

ah, and we (sourceport authors) still can't even find a common ground for state machine scripts, to allow creating cross-port mods...

 

and even if we all will implement that thing, it will still be hard to convince mappers to use it. the beauty of Doom mapping is that you can draw almost anything, and it will render. maybe not perfectly, but will render.

 

and even if we will convince mappers, all maps made before that "quantum leap" won't magically fix themselves. ;-) so in the end of the day, we'll simply have to support two code pathes instead of one, and once mappers will start to use new feaures, neither path could be dropped, they'll be there forever to maintain. so i believe that it is better to try and make the current code faster -- this will not require any discussions, and will improve rendering for all maps instead of only for new ones.

 

so from my PoV, it is both hard to do for community, and maybe even harder for sourceport devs. even GZDoom has only so much more-or-less active developers, and it is the most poplular Doom sourceport out there! others usually have 1-3 devs (and 3 is really ALOT of people), working in their spare time. most sourceports are so different internally, that it would be impossible to create a common code that can be shared among them all. so each developer will have to write that thing almost from the scratch, and then maintain it. even if somebody will send the patch, maintaining that code is still the duty of the "regular" devs. this is hard even for full-time employees, and i think is nigh impossible for hobby projects. alas.

 

4 hours ago, 3t0 said:

So where am I going with this whole thing? Would a new linedef that could act similarly as this HL occluder (toggleable by switches/ACS and what have you), eg turning visibility of whole chunks of level based on player position/triggers, help out?

it is already possible: just use doors! ;-) no, really, for clipper any closed door is the same thing as a solid wall. it is already possible to make windows "close themselves" to hide the interior of the building, and such. just (almost) nobody does that. ;-)

 

4 hours ago, 3t0 said:

While accepting this "offloading onto a mapper" mindset and thinking a lot about this immutability thing since yesterday, yes I could not sleep well, second idea came to me. Why not then just offload immutability calculation onto designers as well?

this basically has the same answer i wrote above. ;-)

 

to get the full advantage of "human factor", we need to completely redefine what "Doom mapping" is. and mappers have to religiously follow all rules, or it will fail. this may work for a brand new game that will use heavily modified idTech1, but... what is the reason to do that? i mean, people will have to learn a new way of mapping anyway, and it won't be much easier than mapping for some modern engine, so why not simply use some modern engine instead?

 

i believe that creating better automatic geometry caching algorithms, better automatic geometry culling algorithms, and so on is the only realistic way now. while we may give mappers some optional tools to use, improving "normal" rendering is the only real way to make engines more GPU-friendly.

 

i, for example, still have a bag full of ideas. it is (theoretically!) possible to use multiple threads and kind of "tiled rendering" for occlusion culling; it is possible to calculate and update rough, but useful PVS in background; it is possible to write better geometry caches, and still upload most of the map to GPU; and so on. maybe none of my ideas will work, but hey, i won't know that if i won't try. ;-)

 

[Mr.Rocket]

8 hours ago, ketmar said:

the idea is neat, yeah. but technical limitations... alas. so translucent objects simply cast no shadows at all, and i don't think that i'll find any way to change this, and still keep the engine realtime. ;-)

 I mean it makes sense and I'm not against it, it is after all an idea and I totally understand.. as my next question is, if it would be possible. ;)

I just wish my damn ears would quit ringing! 

Edited March 11, 2021 by Mr.Rocket

[ketmar]

@Mr.Rocket no worries, i got it. i just tried to explain why it is not possible (at least now), and that i like the idea. i'm really good at explanations, you know, so good that i have to explain my explanations later. ;-)

[Mr.Rocket]

34 minutes ago, Mr.Rocket said:

I just wish my damn ears would quit ringing! 

 :P

in other news...

~ all this means is that someone's (maybe) talking about it.. me or it. nothing more.. in r/l type stuff. ;)

No worries man, you know i throw ideas, but i as well know, that not all, will be accepted.

~ it's common realm of acknowledgement. ;)

Edited March 11, 2021 by Mr.Rocket

[3t0]

47 minutes ago, ketmar said:

it is already possible: just use doors! ;-) no, really, for clipper any closed door is the same thing as a solid wall. it is already possible to make windows "close themselves" to hide the interior of the building, and such. just (almost) nobody does that. ;-)

Now wow! This is really neat hack, that I never realized is possible, thanks for inspiration. I guess with proper texture (pure white or pure black) and very fast door closing times (I hope generic doors can set that) this would work exactly as well as HL2 occluders, I guess. Hope one can disable door sound somehow. One more question: would this work even with ACS and by moving ceiling and floor in-place together from a script? I presume "door" is any sector whose ceiling height is equal to floor height, am I right? So just moving ceiling down instantly should be enough to manually clip-off any level part, right?

 

Spoiler

I guess, given this technique works and still nobody knows/uses it, proves that it would be probably very difficult (eg. near impossible) for mappers to make use of "immutable" sector marking I proposed.

 

And yes as you said while this would be great addition for custom "indie" game, it would still require one to maintain their own fork (of some engine and editor at least).

 

Hmm taking into account all that, it seems like it would be equivalent of sisyphos job or fighting with windmills. I have myself have some forked stuff, where I tried some stupid crap and got too "cosmetix" (as you call it) trigger happy. Meanwhile upstream moved forward and the things ended up in state that they are effectively unmergeable - first my changes were just for me only and that means that I cannot push them upstream, because they would be rejected immediately, and second I cannot even pull new crap-in to my fork because merge becomes so hard it's nearly impossible.

 

Thank you for clarifications, this essentially means, that whole performance issue can be concluded as follows: if one wants have "real 3D" with still pixelated linear (perhaps bilinear) texture mapping some static lightning and reap almost unlimited performance benefits of immutable geometry, it's better to start with Quake 1 as a base right away. On the other hand, if one wants to keep very short world building times and fast map design turnaround times, using some advanced doom sourceport is more appropriate way of going about things. One however loses simplicity of top down mapping perspective and doom's vertical dynamism.

 

Hoping that one would be able to do mixing and matching any of the two concepts together in one engine is given "reasons" probably a pipedream.

 

Finally, I noticed, that in GZDoom, when one uses a MD3 model, it is by default two-sided, eg. each tris normal faces both forwards and backwards, despite tris being single sided in and showing as such in any normal MD3 editor or ioquake3 engine. I roamed through zdoom wiki, but never found a way to get normal 1-sided model rendering working. I have question: is proper MD3 rendering (single sided) supported in k8vavoom?

[Mr.Rocket]

@3t0, I'll push out a simple form of Doom geometry in means to replicate a Quake1 world if you'd like? 

It won't be perfect, but would be the base concept, however would be much so in short a reincarnation.

~ though DM levels heh ~ as I don't have time to start making models.

Similar to a cover band. It will feel and sound like it, but will not and will never be exactly IT. ;)

Though in some aspects, may sound better, you know what I mean...

Edited March 11, 2021 by Mr.Rocket : sorry my typo's :D

[ketmar]

8 hours ago, 3t0 said:

One more question: would this work even with ACS and by moving ceiling and floor in-place together from a script? I presume "door" is any sector whose ceiling height is equal to floor height, am I right? So just moving ceiling down instantly should be enough to manually clip-off any level part, right?

yes, yes, and yes. ;-) clipper simply checks sector height, and if it is zero -- this sector is a closed door. also, most (all?) clippers will clip with solid midtex (think of E1M1, room with the acid: there's secret with a one-sided wall there; the secret passage won't be rendered, because midtex is solid, and completely covers the hole). so there are at least two ways to create clipping planes: doors and 2-sided walls with solid midtex. midtex hack will allow mosters to keep walking around, and won't block sound propagation (zero-height sector blocks propagation unconditionally).

 

yet i doubt that many people are aware of midtex trick. i mean, they mey not know that it helps renderer alot.

 

8 hours ago, 3t0 said:

it's better to start with Quake 1 as a base right away.

i'd say, Quake 3. this is the engine where Carmack finally got everything right (especially collistion detection), yet the engine is still simple, and you can trace its roots back to Q1 if you want to.

 

first Unreal is great too (way better than Quake 1, i'd say). of course, we have never seen its code, because it would be a copyright infringement, but friend of a friend of my friend told me that it is very clean, nicely written, and contains alot of interesting ideas.

 

8 hours ago, 3t0 said:

Hoping that one would be able to do mixing and matching any of the two concepts together in one engine is given "reasons" probably a pipedream.

this makes me sad too. yet, this has a good side: i have some ideas about full realtime map geometry modification (think of full-featured in-engine map editor). if i'll be able to implement that idea (it is still in early reseach phase), Build will run away in tears! ;-) (but don't hold your breath, people; this is one of those ideas that may not work as i expected, and there are no estimates, of course.)

 

i mean, the engine already has a solid low-level framework to deal with map geometry changes, so it is much easier (less work) to add the missing parts than in Quake, for example.

 

8 hours ago, 3t0 said:

I have question: is proper MD3 rendering (single sided) supported in k8vavoom?

i'm almost sure that k8vavoom does it "right". "almost" -- because model rendering code is one of those parts where i hadn't do alot of work, so my knowledge of it is not as good as of some other parts. but as far as i remember, Vavoom is using the usual backface culling mode for models, no black magic tricks there.

 

 

p.s.: people creating such "web visual editors" as this one should be decapitated. with a guillotine created by another "web developer". and with live streaming. and we'll watch that show for decades, because that guillotine will be able to do alot of things except the thing it was designed for. (if you guess what happened... i tried to create a proper partial spoiler. as you can see, there's no spoiler here.)

[jval]

8 hours ago, 3t0 said:

Finally, I noticed, that in GZDoom, when one uses a MD3 model, it is by default two-sided, eg. each tris normal faces both forwards and backwards, despite tris being single sided in and showing as such in any normal MD3 editor or ioquake3 engine. I roamed through zdoom wiki, but never found a way to get normal 1-sided model rendering working. I have question: is proper MD3 rendering (single sided) supported in k8vavoom?

 

3 minutes ago, ketmar said:

i'm almost sure that k8vavoom does it "right". "almost" -- because model rendering code is one of those parts where i hadn't do alot of work, so my knowledge of it is not as good as of some other parts. but as far as i remember, Vavoom is using the usual backface culling mode for models, no black magic tricks there. 

 

K8Vavoom is using culling for models, at least for md2 models (so why not for the others ?).

I figured it out when making the md2 model export for DOOMROCK. I made the export procedure, saved a couple of md2 models, and they were working perfectly with GZDoom & DelphiDoom, but in K8Vavoom only the half triangles were displayed (guess what: it does culling). I had a busy afternoon making all the tris exported properly :)

[ketmar]

3 minutes ago, jval said:

K8Vavoom is using culling for models, at least for md2 models (so why not for the others ?).

yeah, all models are converted to internal format, and i'm pretty sure that it doesn't even have a flag to turn off backface culling. sorry, it is still easier to write forum posts instead of checking the source code, so thank you for confirming it! ;-)

[Gez]

16 hours ago, 3t0 said:

So I finally realized, that unless one is willing to maintain their own full fork (there are few such corpses on github cemetery) it's essentially impossible to change the movement code in any major way (unless some genius of course comes up with a way how to do it somehow, which I doubt).

The only way to do that would be (I apologize for OOP buzzwords) to fully encapsulate movement into a module so that you can have another module with different movement code, and the ability to switch from one module to the other as needed, allowing classic levels to stay with classic movement while a new indie game could try the new movement.

 

16 hours ago, 3t0 said:

Hmm, so even if vertices are bound to their position by BSP rigidity and unable to move on X,Y (topdown) plane, the Doom geometry is still not really immutable, because Z-height can change at any time (lifts, doors - hmm, this is neat hack though, it allows to "reuse" BSP tree for any height :) ) - and yes we are ignoring polyobjects for now, which can move on XY axes, but don't have Z dimension - talk about limitations!).

As far the renderer is concerned, polyobjects do have a Z dimension. You can actually move them up and down. Sure they do not have horizontal surfaces (well, except in SRB2) but their vertical surfaces can be pushed upward and downward. I'm not getting into janky collision detection issues here, that's outside of the renderer's purview.

 

16 hours ago, 3t0 said:

Hmm at first, I thought you just need to analyze sector-linedef "action links" and you are all set, but then I realized with ACS (and other scripting) you can change any sector height at any time right?

The main problem here is stair builder functions. Everything else only affects tagged sectors or the back sector of an active line, and ACS cannot directly target sectors by their number, only by their tag. So if there weren't stair builder specials in the game, you could just decide to "immutabilize" (if you'll forgive this neologism) every sector that isn't tagged and isn't the backsector of an active line.

 

But with stair builders, only the starting step is tagged. Then the stairs recursively propagate through its linedefs. So it's not just any tagged sector and any sector that's the back side of an active line, but it's also any sector that can be reached by crossing linedefs front-to-back from a potentially active sector. At this point you generally fall back into "might as well be the entire map". And this is why you're tilting at the wrong windmill here:

16 hours ago, 3t0 said:

Let me elaborate: with vanilla levels, immutability is a given and mutability detection is simple: everything that has action link (door, lift) is mutable, everything else isn't, period (there are probably few special cases, but this is manageable as you said.

Stair builders are vanilla. Advanced scripting effects aren't what makes this autodetection troublesome (because they rely on tags), this little-used vanilla feature is the real culprit.

 

 

 

 

That said, if you really want to get the benefits of Immutable Geometry on your maps, there actually is a way. Keep your Doom geometry very simple -- no little border detailing, wall indents, etc. Basically keep it in a graybox stage where it handles collision and that's all. The use models to provide the detailing. Your floor models are just 0.01 unit above the "real" geometry (to avoid Z-fighting) and show a more detailed floor. Your wall models are just 0.01 unit away from the walls and they provide all the little details you may want. You would stilll need to double-up your walls (an outer one-sided layer for occlusion purposes, and an inner two-sided untextured layer for collision purpose, with your wall model sandwiched between both) but given how detail-heavy maps can feature dozens of layers per wall, you'd still save up. This massively simplifies the "dynamic" geometry while allowing a level of detail that'd be very hard to do in traditional Doom editing (go on, sculpt some organic-looking tentacles creeping along a wall, have fun playing with the hundreds of minute sloped 3D floors you'll need for that).

[ketmar]

9 minutes ago, Gez said:

So if there weren't stair builder specials in the game, you could just decide to "immutabilize" (if you'll forgive this neologism) every sector that isn't tagged and isn't the backsector of an active line.

i may be wrong here (the code is HUGE, and it is a mess), but isn't it possible to modify the sector that contains a thing with some TID, or a front/back sector of activator line from ACS? it may not be in the original ACS, but i'm almost sure that i've seen something like that in ZDoom extended ACS.

[Gez]

Maybe? I won't pretend I know or remember every ACS function out there. I don't think modifying a sector around a tagged thing is possible, but maybe it is.

[ketmar]

@3t0 see? people still recommend to abuse vanila rendering bugs instead of using Boom specials. there is simply no way we could convince mappers to use something that will make the lives of sourceport devs easier!

[MattFright]

I've tried k8vavoom yesterday after eyeing it for a good while now, and while i was impressed and even have a feeling i could end up playing on it a lot, i felt like the maximum mouse sensitivity you can set it to on the mouse options menu isn't anywhere near high enough for me and my 5$ mouse. I was wondering if you could increase the range of that option for people like me who have to play Doom with a cheap piece of transparent plastic :P

 

Though for the time being... is there any way to push the mouse sensitivity over 10 through the console? I tried what i thought would've been the obvious ways to do it (IE "sensitivity 15", "mouse_sensitivity 15", "mousesensitivity 15", etc.) but none of them worked.

[ketmar]

9 hours ago, MattFright said:

is there any way to push the mouse sensitivity over 10 through the console?

yep. as sensitivities are separate for x and y:

mouse_x_sensitivity 20

mouse_y_sensitivity 20

there is also autocompletion in the console. you can type "mouse" and press <tab> to see all available commands (or have current one typed for you, if it is unambiguous).

 

as for increasing menu values... dunno, you're the first who asked, actually. i mean, sure, i can, it is as hard as changing two numbers, but i feel that the problem may lie in the way i'm working with the mouse. it is quite archaic method, and while it works pefrectly for my GNU/Linux system with X11, it was reported to be quite bad on Windows. but alas, i don't have windows system to properly experiment with that.

 

all the text above means that you can try the console for now, and i'll try to find a better solution than simply bumping the range.

 

and thank you alot for trying k8vavoom, and for your feedback!

 

p.s.: if you want to make your hands dirty, you can open "share/basev/common/basepak.pk3" with, for example, SLADE, and change the values in "uidef/menus/mouse.txt". it is a very simple text file, you'll see what to change when you open it. you'll have to do it with each new build again (i deliberately made main menus unmodifyable from mods), but the price for this small annoyance is that you can do any changes you want without waiting for my approval. go wild (but don't forget to backup the original files ;-)!

 

p.p.s.: for most options in the menus you can press "?" (on US keyboard, with shift ;-), and the engine will print the corresponding cvar name to the console. also, pressing "backspace" will reset the option to default value.

Edited March 16, 2021 by ketmar

[ketmar]

in the meantime, i am working on improving polyobject support. original Vavoom implemented polyobjects by "unofficial Hexen specs" ;-), i.e. "only convex, only in convex sector, never cross sector boundary", etc. since then, advanced sourceports removed most of the restrictions, and i only made k8vavoom don't crash on "strange polyobject usage". so i finally reworked the code, implemented proper clipping and rendering, and so on (and cleaned up rendering code a little by the way). the implementation is not optimised to my usual standards yet, but it seems to work more-or-less ok.

 

i am also experimenting with fully 3d polyobjects (i.e. polyobjects with floor and ceiling). no, you won't be able to stand inside them, but they could be used to implement platforms, for example. and i am 100% sure that if they'll work, somebody will build complex compound object out of them, to make me regret that i introduced that feature.

 

the idea is that polyobject made of 2-sided lines will behave like a 3d floor, only freely moveable in all directions (and the flats can't be sloped). i am still in research phase, so the feature may end up scrapped (like many other ideas i had), but hey, trying stupid things is fun!

 

just please, don't use the feature even if i'll implement it. i HAET platforming sequences in 3d games!

[ENEMY!!!]

That sounds potentially exciting to me.  I'm aware that if 3D polyobjects get implemented, some might be tempted to use them to increase the emphasis on "jumping puzzles" and the like, which especially with Doom's physics probably wouldn't end well, but I can see scope for very positive, imaginative and ground-breaking use of them.  It would be easier to implement things like the impressive train sequence at the start of Heartland (which was done with the Eternity Engine but required the extensive use of portals and elaborate scripting) and horizontally moving lifts, cranes etc, and holes opening up in the floor.  So I'd say it's definitely worth trying it out, though I'm aware that it might not work.

Edited March 18, 2021 by ENEMY!!!