Introducing ZokumBSP

[Graf Zahl]

Edward850 said:

You never acted like this when you discovered ZDoom had the size of the spidermastermind wrong for several years.

No, when that happened, I said "Sorry, I made a mistake several years ago - it's fixed now." No harm was done, no mod got broken for it and no user suffered long term damage from it. Ultimately it was just a harmless bug that had no lasting consequences.

Bugs can happen. Things can break. That's life. This only becomes a problem if a bug gets fixed after such a long time that the mere fix may become a problem in itself. ZDoom had a few of those, too, over the years, and in some cases I had to do things I really would have liked not to do to deal with them.

But this case is even different, it's not so much a fix, it's just a tool that produces data which is plain and simply incompatible with many engines out there that are still in use but won't see a chance of ever getting updated. What, for example would you tell those users who want to hold on to plain ZDoom? Sorry, map doesn't work because I chose to use a node builder that's incompatible? Or those with old OpenGL 1.5 hardware which are stuck with GZDoom 1.8.6? Or those who use PrBoom but refuse to use complevel 2? Or (gasp!) some DOS lovers who are using the latest DOS incarnation of Eternity?
THOSE are the people who will run into problems, not the ones using GZDoom 2.4.0 once it will be released. Those will never notice any problems, you can rest assured of that.

Edward850 said:

There's a rather lot of builds of ZDoom that don't run hellfact.wad properly without rebuilding nodes. You added 3dfloors to BTSX via a compatibility text file to eliminate rendering artifacts of the GL renderer, but then there's all those older versions of GZDoom that don't have that compatibility...

Read my paragraph above. It's one thing to fix bugs in newer versions of an engine, but it's an entirely different thing to make maps that are supposed to run on 'everything' but don't because only the latest 'everything' supports what it comes with.

If you want to promote 'vanilla' as the format every port can play than you should show some consideration to those users who are not on the latest or greatest version of any engine.

Hopefully it gets a bit clearer to do an analogy in the world of Windows operatinmg systems:

So the source port is the operating system, your map is the program, you are essentially saying "This software was made for Windows 95" and is fully forward compatible. But wait, in Windows XP something changed that made the program crash, the bug was discovered and fixed years later, let's say in Windows 8.1. And yet you insist that "It ran on Windows 95 so it must run on any other version, too - I don't care that 3 or 4 intermediate ones had a bug. People should upgrade to Windows 10 anyway." But the reality looks like a significant percentage of users actually using these intermediate versions, so you cannot really afford to ignore that the bug exists.

[Edward850]

What do you tell those OpenGL 1.5 users running an old GZDoom playing the first map of BTSX? Or hell, anything sitting inside compatibility.txt. You keep saying it's completely different, but the only thing that's completely different is what you practice from what you preach.

Graf Zahl said:

Or (gasp!) some DOS lovers who are using the latest DOS incarnation of Eternity?

You mean all of 0 people? Nobody would be using that anymore as their legitimate de facto port.
...
Unless... Graf why didn't you tell us you loved DOS Eternity that much? Someone might have cared about compiling new builds for you.

[Altazimuth]

Eternity now has this fixed. When checking the validity of a blockmap it also checks if any of the starting shorts of a blocklist are nonzero; if they are nonzero (and the demo of the version played is for 3.42.00 or later) then it has vanilla behaviour, otherwise it goes to Boom behaviour.

-vanilla was fine (as pointed out by dew), and nothing has changed there.

[40oz]

Graf Zahl said:

Seriously Edward,

Please ignore him. He always stirs up shit. I'm sure there's a common ground that can be found here, if we talk politely about it without rude/sarcastic remarks.

GZDoom players being able to play the BTSX the team wants to make are pretty high stakes for everyone involved, dont let this shithead ruin everyone's fun and make this a needlessly divisive ordeal.

[dew]

Altazimuth said:

Eternity now has this fixed. When checking the validity of a blockmap it also checks if any of the starting shorts of a blocklist are nonzero; if they are nonzero (and the demo of the version played is for 3.42.00 or later) then it has vanilla behaviour, otherwise it goes to Boom behaviour.

Excellent news! Thanks, altaz and zokum!

[Graf Zahl]

Edward850 said:

What do you tell those OpenGL 1.5 users running an old GZDoom playing the first map of BTSX?

Nothing?
The first map works fine even with older versions of the engine.
As for the map that needed fixing, there's two options: Copy the fix or live with a small visual glitch. It's nothing game breaking, just a few hacks not getting properly rendered. The mod is perfectly playable as it is.

But I see you would release a map to the public that needlessly breaks on some older ports without giving it a second thought. Now who's the inconsiderate one here?

[Edward850]

Graf Zahl said:

But I see you would release a map to the public that needlessly breaks on some older ports without giving it a second thought. Now who's the inconsiderate one here?

Yes, how dare I make a map utilizing improved vanilla capabilities. Only some top-level monster frisbee could ever be capable of such a crime as to make a map with advanced vanilla requirements. Heaven forbid anybody experiments with vanilla Doom in the year of our lord, 2017, least we incidentally expose a bug in some source ports that aren't updated and never accounted for it.

[Graf Zahl]

Let me go back to my analogy.

What will happen if you release a source port that only works on Windows 9x and Windows 8.1 and higher but not on XP, Vista and 7?
Correct: Some people stuck on these systems would grill you for it.

While it may be technically sound what you do, it'd still be utterly inconsiderate. And if you cannot see that I feel sorry for you.

[Edward850]

It's a good thing we are talking about vanilla compatibility of source ports with specs that already provably exist, not sticking Doom on Windows95 of all things, then. Your analogy couldn't be any more forced and off point if you tried.

[Graf Zahl]

Edward850 said:

It's a good thing we are talking about vanilla compatibility of source ports, not sticking Doom on Windows95 of all things, then.
You analogy couldn't be any more forced and off point if you tried.

Actually, the analogy is spot-on.

Your map would work on Doom.exe, it would work on ZDoom up to version 1.12, it would not work in ZDoom 1.14 up to GZDoom 2.3.2, but it will again work in the next version, whatever it will be named.
For other ports there'd be similar windows of brokenness and you are essentially telling everybody using such an engine 'upgrade or die because I was too cavalier to consider your special situation'. (i.e. you ignored their situation out of choice, not out of ignorance and certainly not out of necessity.)

[Mordeth]

Graf Zahl said:

What will happen if you release a source port that only works on Windows 9x and Windows 8.1 and higher but not on XP, Vista and 7?
Correct: Some people stuck on these systems would grill you for it.

Not to fan the flames, but that did happen to me when GZDoom switched from 1.x to 2.x. and my old-ass no-longer-supported graphics driver could not cope. Grilling did not happen.

[esselfortium]

For what it's worth, BTSX is probably going to be fine without needing the "leading 00" optimization. Yes, it would allow for even more blockmap squeezing, but I'd rather avoid breaking ports if we can avoid it. ZokumBSP includes other blockmap optimizations that AFAIK are port-compatible already, and should hopefully still help provide a bit of blockmap breathing room on our larger maps.

I am happy to see this fix being implemented in ports, though! The bug is an obscure historical oddity that I don't think any of the port developers active today could have possibly known about, so I don't think there's a reason to throw blame around for it -- it's just an old oversight that can be fixed. (And it's a relatively easy fix, according to Altazimuth!)

With that said, I don't think it's too big a deal for users to need an updated version of their port of choice in order to run a new wad. I think BTSX E1's first release even included a note recommending that Eternity users update to the latest version to ensure some port-specific definitions worked as intended. And I'd expect that most users of (G)ZDoom, especially, are already accustomed to needing to install a bleeding-edge version every time a new wad comes out.

[Wagi]

Out of curiosity, do we know exactly why ignoring linedef #0 in blocks that shouldn't include that line breaks demo compatibility? In theory, if the line is outside of the block's boundary, it shouldn't matter whether or not it's listed in the block, right? Is this one of those occasions like E3M6 where we get overflow depending on distance and positioning, or is it something else?

[Linguica]

Just as one example, if a player is sliding along a wall, they could get an "elastic collision" with linedef 0 under certain circumstances even if the line is on the other side of the map.

[VGA]

I don't understand all this bruhaha. If the BTSX team would use this option then they could provide a secondary non-vanilla download.

Also, can't users easily fix any such mappack by using a node builder? Or maybe if some lumps are deleted with SLADE then ZDoom uses its internal node rebuilding function?

It's not like there are many mappers that are creating non-limit-removing vanilla maps which are at the edge of what the format can handle. So people will not be using this extra option, it's not the default from what I understand.

[Rachael]

IMO: If source ports break because of the MBF fix, later versions of said source ports can include a fix that either generates their own nodes or stops ignoring the leading 0-entry at the beginning of every blockmap.

If the latest versions of such ports no longer have a maintainer, we as a community can come together (since it's open source anyway) - provide fixes to such ports, and release our fixes to said ports unofficially. (I'm thinking in particular of ZDoom, Skulltag, and Vavoom, in this case) - although arguably, if anyone is using the ports I listed they probably really should move on to alternatives (like GZDoom if using ZDoom, Zandronum if using Skulltag, etc).

For those that do have active maintainers, a point release for this fix may be appropriate.

Will it suck having so many old versions of these ports not working with the newest BTSX version? Yes. But it's certainly not going to kill anyone.

[Lollie]

I've tried to keep up with this thread, but the frustration and finger-pointing surrounding this is so mindboggling. I don't remember seeing this level of anger when linguortals were discovered/demonstrated as a vanilla-compatible effect, despite it breaking in modern ports. I'm not seeing how this is any different - something that is perhaps unintended by the original developers, but compatible with Doom.exe nonetheless. It ought to be a prime opportunity for learning new things about how the Doom engine works.

And c'mon. People on previous pages arguing for compatibility with old versions of ports? Even ZDoom 2.X implemented features that modern maps make strong use of, stuff that would never work in a 1.X version of ZDoom. Legacy compatibility can only be upheld for so long. Let old versions go obsolete, discontinue their use and move onto newer, currently-maintained ports. No port lasts forever.


edit: Just now noticing Essel's post, so much noise in the thread caused me to skip over it. If forgoing the 00 optimization isn't going to be a big problem, then it sounds like a good compromise. It'd be great to know more about these new optimizations though, it's interesting that this has gone overlooked for so long.

[Linguica]

3noneTwo said:

It'd be great to know more about these new optimizations though, it's interesting that this has gone overlooked for so long.

For the sake of historical accuracy, most (if not all?) of the blockmap optimizations have been proposed at one point or another in the past; here's fraggle listing out a bunch of ideas back in 2013.

[Graf Zahl]

3noneTwo said:

And c'mon. People on previous pages arguing for compatibility with old versions of ports? Even ZDoom 2.X implemented features that modern maps make strong use of, stuff that would never work in a 1.X version of ZDoom. Legacy compatibility can only be upheld for so long. Let old versions go obsolete, discontinue their use and move onto newer, currently-maintained ports. No port lasts forever.

It's a major difference to make a map for a new port, using said new port's current features and advertising it for said modern version than making a map that's supposed to run 'everywhere' but won't do precisely that due to some ill-conceived decision from 18 years ago that wasn't factored into the decision-making.

@Eruanna: Yes, you can do 'unofficial' point releases to abandoned engines that may still be relevant - but the question is, will people actually use them?

I'm sorry to say this but this entire discussion strongly reeks of insider bias, i.e. many people projecting their own knowledgeable view of the matter onto the general public. The thing is, the general public of Doom players doesn't think like us insiders - they are not likely to permanently upgrade their engine of choice, and many do run on older versions because they are 'good enough' for their needs and they do not stay informed about such technical details.

I am 100% convinced that none of the forum users here and at ZDoom will have any problems dealing with such maps, but come on! How many percent of Doom players are actively posting here? 5%, 2% or even less? And how many are constantly upgrading their engines? Not many. The only number I have to throw around is GZDoom downloads, but I think it should be enough to make my point:
2.1.1/1.9.1 got 170000 downloads, the current 2.3.2 sits at 28000 downloads right now. Granted, there's certainly a large amount of atrition to be counted here, but let's assume that half of those 170000 downloaders are still active. That's 85000, of which at most 28000 have upgraded to the latest version, meaning almost 60000 users who are running an old version and seeing no reason to upgrade, and that doesn't even consider all the uncounted ones with more obscure choices.
THESE are the people who would be affected by some vanilla maps appearing that do not follow the established specs from 18 years of history, regardless of how this relates to vanilla compatibility.

[Rachael]

So BTSX E2/3 should include a disclaimer:

"Will not work on ZDoom 2.8.1, GZDoom 2.3.2, Eternity [whatever latest version now], Vavoom [etc]. Please download the [blockmap fix versions] of these engines if you wish to use them with this mod!"

And if a ZDoom 2.5.0 user doesn't read that disclaimer and wants to try BTSX E2 - well that's entirely his own problem.

[Graf Zahl]

The main question should be: Does a map even NEED such a compatibility breaking blockmap? In my opinion this should be a last-ditch optimization setting that only gets activated if without it the blockmap gets too large, and then the node builder should output a warning.

[Rachael]

Well that is a fair point. It doesn't really NEED it.

[Altazimuth]

Well at the moment the optimisation isn't required for the maps we've seen, but it seems wise to work alongside Zokum to prepare our ports for these optimisations; so when the time comes that there are maps that need these currently-breaking optimisations, advanced ports can be ready to play them correctly.

I personally welcome the challenge.

[dew]

Well, the mission statement is literally "to enable the construction of bigger maps without losing compability with the classic DOS executables and Chocolate Doom". zokum has been using Misri Halek as the motivator for the project (which had to be carefully hand-optimized by Malde) and I mention two maps that are also right at the blockmap limit in the OP. You can check them out, they're e2map20 and e2map25. Add a small sector outside of the map's bounding box and they're going to explode. Mechadon's Unstable Journey is stabilized (I'm deeply sorry for the pun) and fully playable, because the man behind is crazy like that, but map20 has suffered from not being able to add even simple monster teleclosets anywhere anymore and a major rehaul was planned to trim it down/restructure it in order to have more "functional" space. There might be another E3 map that might have even worse problems, maybe.

As essel said, we'll try to milk any optimizations we can without having to open the linedef 0 can of worms and hopefully we'll get a nice buffer for extra structures anyway. The trick does affect too many ports. Zdaemon would probably get a fix out in order of years, heh.

As a side note, compatability with old *Zdoom versions is really low on the priority list. Updating to the latest nightly has been the unofficial #1 step to solving any issues for years. And we're not targeting, say, Doom2 v1.666 compatability either.

VGA said:

it's not the default from what I understand.

Actually it is right now, but zokum will change that in the next RC and it'll have to be set on manually with a flag.

[Graf Zahl]

dew said:

Zdaemon would probably get a fix out in order of years, heh.

That may happen - but with ZDaemon it's always hard to tell, considering it's closed source.

dew said:

Actually it is right now, but zokum will change that in the next RC and it'll have to be set on manually with a flag.

That's good to hear and is essentially what I was advocating for all about. If it can build optimized blockmaps without that compatibility hazard, the tool will be a lot more valuable.

dew said:

As a side note, compatability with old *Zdoom versions is really low on the priority list.

Don't laugh, but I remember one person a few years back who stated that he was using 2.0.63a 'because it was the last good version of ZDoom' and he saw no point upgrading ever. As stupid as it may sound, these people do exist.

dew said:

Updating to the latest nightly has been the unofficial #1 step to solving any issues for years. And we're not targeting, say, Doom2 v1.666 compatability either.

I sincerely hope that this will stop with a more frequent release schedule.

[Marcaek]

Graf Zahl said:

Don't laugh, but I remember one person a few years back who stated that he was using 2.0.63a 'because it was the last good version of ZDoom' and he saw no point upgrading ever. As stupid as it may sound, these people do exist.

There are people who refuse to use Zandronum because they consider it a "downgrade" from Skulltag. Ask them what they mean and you won't get a legit answer.

Don't cater to fringe loonies.

[Graf Zahl]

Generally I agree but these people unfortunately also have the tendency to spread false information quite liberally if things don't go their way.

[zokum]

One of the goals was that the executable should be a slot-in replacement for ZenNode. My definition of that is that given the same parameters, it would build maps that are just as good as the one ZenNode built, and hopefully better.

My goal was vanilla doom2.exe compability, and I didn't check much with ports, apart from Chocolate Doom. This is mostly due to me using demo playback to verify the integrity of the blockmaps built.

Producing blockmaps with the 00-header will be the default option in the next released version available on the web site, simply due to so many ports having a faulty optimization that relies on this behaviour.

If a wad targets a specific version of Doom like vanilla/chocolate or newer EE, they could safely turn 00-headers off if they need it for their project to stay within the limit.

I was aware of the thread from a few years ago, and I plan to implement things from there as well as new things not mentioned in this thread.

1.0.9 will most likely be released with a new and novel optimization that shaves off a few bytes on most maps. I tested it on av.wad and doom2.wad and it found something to compress on every map.

This optimization is the type of optimization that the mapper can use to his/her advantage and add more detail to maps without worrying about hitting the limit.

Here are some very prelimenary numbers from av.wad:
MAP01 BLOCKMAP - 4184/4216 (99.24%) Compressed: 94.92% 0.014 secs
MAP02 BLOCKMAP - 4382/4446 (98.56%) Compressed: 93.35% 0.014 secs
MAP03 BLOCKMAP - 14398/14502 (99.28%) Compressed: 72.12% 0.262 secs
MAP04 BLOCKMAP - 13398/13690 (97.87%) Compressed: 74.31% 0.293 secs
MAP05 BLOCKMAP - 8382/8392 (99.88%) Compressed: 85.65% 0.037 secs
MAP06 BLOCKMAP - 15988/16092 (99.35%) Compressed: 71.52% 0.418 secs
MAP20 BLOCKMAP - 53062/53148 (99.84%) Compressed: 87.57% 1.508 secs
MAP21 BLOCKMAP - 11746/11988 (97.98%) Compressed: 95.40% 0.017 secs

The old code gave me a blockmap of 4216 bytes for map01, the new code a blockmap of 4184 bytes.

av.wad map20 with maximum compression: ./ZokumBSP -bz=0o=3sg -n- -r- av.wad map20

MAP20 BLOCKMAP - 51310/53062 (96.70%) Compressed: 84.81%187.033 secs

However, further testing is needed to refine the approach and check the compatibility. It will break demo compatibility. It might turn out that this has side effects that make this an optimiation that isn't usable.

To clarify the point about demo compability. It only breaks if you rebuild the blockmap with a different setting for this new option. This is the same as with the 0-header and subset compression techniques.

[Jon]

In the age of Open Source I don't understand the desire to stick to supporting particular old *binary* versions of ports. If enough Joe Bloggers preferred XYZDoom 2.0.whatever for some reason, they could keep that branch alive as source for as long as they wished, and fix blockmaps whilst they're at it.

[VGA]

Does anyone have a sample wad where the problem is immediately obvious?