Introducing ZokumBSP

Linguica said:

For what it's worth, isn't blockmap compression also helpful for source ports? I know the WAD format uses signed shorts for the blockmap index and there are various ports that will accept unsigned shorts, which doubles the available space, but blockmap compression could enable those ports to play even larger maps without an overflow.

That would only apply to ports without their own internal blockmap builder. Any port that derives from Boom will have such a thing, and will automatically discard any blockmap that appears to be too large or otherwise broken. I think of the better known engines, aside from the deliberately conservative ones, Doom Legacy is the only one without its internal blockmap builder, so for mapping to an enhanced standard the blockmap has been mostly irrelevant, most ports won't have a problem even if it was completely missing.

Edward850 said:

The whole point of this BSP builder is research into highly optimized/compressed vanilla nodes, and the entire point of a vanilla map is it works in vanilla specifications. How is this Zokums fault if ZDoom or other source ports doesn't properly abide to those?

What is "vanilla specifications"? Where is the document? Zokum found out that something that had been overlooked for over 23 years can be done and suddenly it invalidates every port since MBF? Get real.

AlexMax said:

I've been trying to word this next sentence in a way that doesn't make me sound like a pedantic jerk, but I've given up, so please forgive me:

A vanilla map, by definition, works in vanilla doom2.exe.

You're not a pedantic jerk for making that dumbass point; you're a stupid one.


The overwhelming majority of people who are going to play BTSX are not going to use the vanilla doom2.exe for it.

Gez said:

You're not a pedantic jerk for making that dumbass point; you're a stupid one.

The overwhelming majority of people who are going to play BTSX are not going to use the vanilla doom2.exe for it.

What are you contesting here? AlexMax is correct. Vanilla wads are meant to be compatible with the original unaltered Doom and Doom 2 executables. Whether people actively use doom.exe (or vanilla-limiting ports) to play them is irrelevant.

People play Doom via DOSbox. The official Steam and GOG releases make this incredibly easy.

Gez: The specification I go after is the DOOM source code. It has been known for a long time that nodebuilders output a format that isn't what the engine actually expects. The error is present in the released idbsp source code, which was released very early, around 94-95 or so. The first node builders produced data similar to what the iwads had, not knowing that it was in facet incorrect from an engine point of view.

Currently, all nodebuilders i know of, apart from ZokumBSP, output an extra 00 header which is then ignored by a lot of ports or slows down the engine. What Lee Killough did is a bad optimization. His code should have checked if there are unneeded 00 headers, if so, ignore them. If they're not there, don't just blindly ignore the first bytes.

What the doom scene is currently doing is creating a bloated blockmap with 00 entries that a lot of engines then ignore. The correct fix would have been to fix the nodebuilders when this was realized /confirmed when the source was released. One shouldn't assume that all future wads will also be built in an engine-incorrect way and then optimize it away runtime.

There is a special kind of insanity in adding known bad blocks of data to a highly size constrained data lump and then ignoring this unnecessary data in most of the programs that use this lump.

The 00-ignore-optimization also throws away demo compatibility. ZokumBSP no-0-header-blockmaps are only a problem in ports that aren't demo compatible with 'vanilla'. The ports that can play back most such demos, which any decent port should be able to, should be able to play even the highly optimized blockmaps just fine.

There are some demos that can be tricky to play back correctly, since they rely on engine bugs and possibly certain memory constraints to play back correctly. This is a problem with the algorithms in the original engine and fairly hard to emulate.

If one is aiming for compatibility with broken ports that can't be fixed by settings/ command line switches, simply turn on the header generation in ZokumBSP and enjoy the other optimizations that are still possible with the tool.

If I make a wad that is GZDoom only, I don't consider it a defect that it doesn't run properly in EE or Legacy etc. Same with this tool. Let the mappers decide.

This tool can produce blockmaps that can be handled in, as far as i have had reported: prboom-plus, chocolate doom, vanilla doom, eternity engine. Some of them require vanilla behaviour "fixes" to be turned on. Personally I mostly tested the code with chocolate doom, a very nice port.

Just popped up to say thanks to Zokum for making ZDoom fans butthurt.
Thanks a bunch Zokum! (and for messing with Doom nodebuilders in 2017, too.)

Gez said:

The overwhelming majority of people who are going to play BTSX are not going to use the vanilla doom2.exe for it.

Weak and wrong argument. BTSX is meant for vanilla. And there are high resolution ports available too, that emulate vanilla, so there is no excuse.

3noneTwo said:

What are you contesting here? AlexMax is correct. Vanilla wads are meant to be compatible with the original unaltered Doom and Doom 2 executables. Whether people actively use doom.exe (or vanilla-limiting ports) to play them is irrelevant.

Yeah yeah yeah. Whenever someone asks what format they should try mapping in, the answer will be something along the line of "vanilla if you want your map to be compatible with every port", not "vanilla if you want your map to be played in the original exe".

Look at the argument for Freedoom going vanilla, it was "compatibility with all ports", not "compatibility with the original doom2.exe".

3noneTwo said:

People play Doom via DOSbox. The official Steam and GOG releases make this incredibly easy.

People who play Doom in DOSBox through the easy Steam or GOG shortcut do not load any PWAD. People who know are savvy enough to know about Doom mods also know about source ports -- and generally use them.

printz said:

Weak and wrong argument. BTSX is meant for vanilla.

Then why does the text file mention it has also been tested with Eternity Engine, PrBoom-Plus, ZDoom, GZDoom, and Risen3D? Then why does it contain MAPINFO lumps?

It's meant for vanilla and ports, not for vanilla alone, playing with ports is forbidden.

// Revision 1.3  1998/01/30  23:13:10  phares
// Fixed delimiting 0 bug in P_BlockLinesIterator

Gez said:

Then why does the text file mention it has also been tested with Eternity Engine, PrBoom-Plus, ZDoom, GZDoom, and Risen3D? Then why does it contain MAPINFO lumps?

It's meant for vanilla and ports, not for vanilla alone, playing with ports is forbidden.

Not only that. Both BTSX mods also contain some beautiful tall skies so that it looks even better in ports that have freelook enabled. Actually, I was quite surprised when I realized that.

In the end this entire 'map for vanilla if you want your map to run in all ports' argument is totally bogus. Aside from the very few ports that 'stay true' to vanilla to the detail there isn't one single engine out there which is not at least limit removing, and even here, Boom support is at least in the works for all of them.

Here's a quick example of a map that's going to be unplayable if GZDoom "bruteforces" the solution. Wouldn't it be funny if new vanilla maps started using it just to spite a certain someone?

On a more serious note, I find it a little ironic and a little more sad that such exciting progress and optimization is getting aggressively shouted down by the lead developer of the most advanced source port. I understand it doesn't benefit GZDoom all that much (if at all), but the bullying campaign on display in this thread is very cynical. You're just defending what's yours. I particularly dislike turning our effort to make BtSX as inclusive and user-friendly as possible against us.

If it weren't for the limit-extending nature of this tweak (allowing use of more space within the blockmap) I'd dismiss these arguments about vanilla maps being for "vanilla only" off-hand as pedantry, legalism, and pointless semantic argumentation of the type that would make William James spin in his grave, but since that facet exists I'm willing to give the whole thing a fair shake and make sure Eternity tolerates this type of blockmap.

When ports like mine spend decades of effort to maintain vanilla compatibility within reason, and then people come along angrily red-face shouting some bullshit about maps that will only run in vanilla like it's some form of virtuous comeuppance to all of us port developers who have committed the sin of trying to provide a more stable predictable experience for mapping and gameplay, I wonder what the fuck is wrong in their brains. Pick some other kind of battle to fight, seriously.

dew said:

On a more serious note, I find it a little ironic and a little more sad that such exciting progress and optimization is getting aggressively shouted down by the lead developer of the most advanced source port. I understand it doesn't benefit GZDoom all that much (if at all), but the bullying campaign on display in this thread is very cynical. You're just defending what's yours. I particularly dislike turning our effort to make BtSX as inclusive and user-friendly as possible against us.

What kind of weed have you been smoking? Where have I said ANYTHING bad about BTSX, huh?

dew said:

On a more serious note, I find it a little ironic and a little more sad that such exciting progress and optimization is getting aggressively shouted down by the lead developer of the most advanced source port. I understand it doesn't benefit GZDoom all that much (if at all), but the bullying campaign on display in this thread is very cynical. You're just defending what's yours. I particularly dislike turning our effort to make BtSX as inclusive and user-friendly as possible against us.

Pointing out that it will break many of the source ports out there is hardly bullying. It is not Graf's fault if you don't like this inconvenient truth.

3noneTwo said:

What are you contesting here? AlexMax is correct. Vanilla wads are meant to be compatible with the original unaltered Doom and Doom 2 executables. Whether people actively use doom.exe (or vanilla-limiting ports) to play them is irrelevant.

True, but these days, so what? When did Carmack release the source 1997? For almost 20 years there have been source ports and for that entire time this quirk has gone unnoticed. Now, while I'm not saying that finding out new stuff about Doom isn't fun and interesting (hell, we even have a thread about "things you just found out") doing something that means ports can't play maps that they otherwise would have simply because of a newly discovered vanilla quirk almost 20 years after source ports first appeared and which no one has used ever until now?

AlexMax is "technically correct" (which Futurama tells me is the best kind of correct) in as much as Vanilla means "the vanilla exe can play it" but the practical application of the term is that the specs that people have understood and created maps, tools and ports to for the last 20-24 years are Vanilla. So when a new thing comes to light after all that time that makes ports which have supported the "vanilla standard" for all that time balk at a map, it could be argued that such a quirk, even if it runs on the Vanilla exe, is not really in compliance with the de facto vanilla spec.

3noneTwo said:

People play Doom via DOSbox. The official Steam and GOG releases make this incredibly easy.

Yes, but how high are those numbers? Easy? Possibly, but desirable? Best? That's subjective at least (not that you said it was desirable or best). There's no real way of telling how many people play using DOSBox+doom.exe versus people playing using ports but it won't be the majority; I'm pretty sure of that. Like I said, the source has been out for almost 20 years. Doom was on its own as the vanilla exe for 4 years. Source ports have been available for a long, long time. Hell, some source ports have been around for longer than the people who play them have been alive.

I'm not saying that people shouldn't play doom.exe (whatever rows your boat, it's all good IMO) but people who still hold to the idea that it is the right, proper and only way to play (especially those who crusade about it) don't seem to acknowledge that situation ceased some time ago. What's more, it happened with Carmack's blessing, nay, encouragement.


However, I now stand corrected that the type of blockmap under discussion doesn't have to be built by zokumBSP.

dpJudas said:

Pointing out that it will break many of the source ports out there is hardly bullying. It is not Graf's fault if you don't like this inconvenient truth.

It's only really inconvenient for ports that ignore proper vanilla compatibility, only meeting arbitrary definitions of it, though.

Edward850 said:

It's only really inconvenient for ports that ignore proper vanilla compatibility, not just arbitrary definitions of it, though.

By the end of the day, it doesn't make a difference. Either your map runs on a port or it does not.

The part which you now use to just gang up on me oh so gracefully was reacting to this:

Gez said:

Then why does the text file mention it has also been tested with Eternity Engine, PrBoom-Plus, ZDoom, GZDoom, and Risen3D? Then why does it contain MAPINFO lumps?

It's meant for vanilla and ports, not for vanilla alone, playing with ports is forbidden.

BtSX is a vanilla megawad and we were excited about this new tool that would help us expand some maps that couldn't grow any larger anymore and perhaps even suffered from it quality-wise. Testing with a wide variety of ports is what brought the issue to light in the first place and it's why I moved the thread into public. The sheer saltiness and aggression that's going on in here is making me regret it.

The irony and sadness comes from the fact we'll probably have to limit ourselves because of the advanced ports and ignore the linedef 0 optimization flag, because we'd compromise the user-friendliness goal.

dpJudas said:

By the end of the day, it doesn't make a difference. Either your map runs on a port or it does not.

Yes, that is indeed the exact definition of compatibility. If you have a port that claims to be vanilla compatible, you can't just half measure it.

dew said:

Here's a quick example of a map that's going to be unplayable if GZDoom "bruteforces" the solution. Wouldn't it be funny if new vanilla maps started using it just to spite a certain someone?

The map is broken. Period. It'd fail any blockmap verifier that wants to ensure that all lines are present. It's also fail all engines that do not use the original blockmap at all. And there's no reason to make such maps aside from wreaking havoc.
It'd even fail in a port like Eternity if someone was to force a blockmap rebuild. Sorry to burst your bubble, but I think one of the absolute requirements of a proper map must be that it survives a blockmap rebuild, because you have absolutely zero control over people actually using the blockmap that's in the file.
Bottom line: Nothing to worry about, aside from a mapper who'd get into a shitstorm by the users who couldn't play the map.

Edward850 said:

It's only really inconvenient for ports that ignore proper vanilla compatibility, only meeting arbitrary definitions of it, though.

Wrong. This thing is only inconvenient for users of older versions of source ports which cannot retroactively be fixed. But this seems to go over the head of some people here.

Its cool that this kind of thing exists, but I think mappers have to be cognizant of the kind of mapping projects they want to release. Sure you can call your project vanilla under the exact technical definition, but you do have to realize that the thousands of Doom players have a tendency to ambiguously assume that vanilla translates to "will work on everything."

If you're making a vanilla project and intend to use this, the risk of breaking compatibility with advanced source ports is there, so that's a conscious decision you have to make. Just as people who make mods for advanced source ports have to alienate people who prefer to play vanilla doom, apparently theres a new level in which making something that's too vanilla (for lack of more sophisticated wording) will alienate people who prefer advanced source ports. I think its a bit presumptuous to assume that source port developers have a responsibility to cater to your personal interests, however easy and miniscule they may be.

So you have to ask yourself, what's more important to you; your rules or your players?

I see the Doomworld mafia is in full swing again, taking totally unjustified potshots at those few who seem to be able to keep a levelheaded view at the situation at large beyond the tiny pathetic niche somewhere back in 1993 in which you all seem to mingle in.

You people are truly a lovely bunch. And dew, your 'example' is really just a pathetic excuse of ill-motivated activism.

dew said:

Wouldn't it be funny if new vanilla maps started using it just to spite a certain someone?

Personally, I'd interpret that as bloody mindedness; typical of the attitude I've seen from some of the Vanilla soldiers in the community over the years.

Presumably the "certain someone" is Graf who has already committed a change to the GZDoom code to handle blockmaps of the type under discussion. So it would be no skin off his nose and the only people hurt by this action would be the ones who wanted to play the map in their port of choice but who couldn't. And, lets be honest, how big a hurt would that be? "Oh gnoes, I can't play that one map, whatever will I do". :P

40oz said:

Its cool that this kind of thing exists, but I think mappers have to be cognizant of the kind of mapping projects they want to release. Sure you can call your project vanilla under the exact technical definition, but you do have to realize that the thousands of Doom players have a tendency to ambiguously assume that vanilla translates to "will work on everything."...

Exactly my thoughts. A well worded post.

I'd suggest that, these days, the de facto vanilla standard is a most-basic set of map parameters, features, editing tricks and so on that will work on any port claiming vanilla compat. Clearly, maps with this blockmap type don't comply with that.

A vanilla map that isn't to vanilla spec. Doom paradox alert. LOL

Hell Theatre said:

I see the Doomworld mafia is in full swing again, taking totally unjustified potshots at those few who seem to be able to keep a levelheaded view at the situation at large beyond the tiny pathetic niche somewhere back in 1993 in which you all seem to mingle in.

You people are truly a lovely bunch. And dew, your 'example' is really just a pathetic excuse of ill-motivated activism.

Meanwhile the ZDoom mafia is in full swing attempting to quash and amount of discovery and development of vanilla specifications, least they have to admit just how far away from vanilla Doom compatibility they actually are.
This happens every time something about the vanilla format is discovered, and it is always exclusively ZDoom users and devs that complain. It comes off as you guys coming here yelling "we're a doom port, we're a doom port!" at us while your port slowly turns into a corncob.

40oz said:

So you have to ask yourself, what's more important to you; your rules or your players?

Bingo! That's the $64000 dollar question that some of the crusaders here seem to forget.

Enjay said:

So it would be no skin off his nose and the only people hurt by this action would be the ones who wanted to play the map in their port of choice but who couldn't. And, lets be honest, how big a hurt would that be? "Oh gnoes, I can't play that one map, whatever will I do". :P

I can tell you what will happen. Enter the comment section on /idgames and there may be a flood of comments along this line: "This mapper is an idiot, intentionally breaking compatibility with most source ports. Avoid!", plus a 0-star rating.

And it won't get further than that. The only person caught in the crossfire of bad opinion will be the zealot himself.

40oz said:

So you have to ask yourself, what's more important to you; your rules or your players?

I hoped source devs might want to explore the issue and discuss how to fix it among ports. Your question wouldn't be a Sophie's choice then. I think it's fairly obvious the willingness for that is... not universal.

I dont feel that it was ignorant of you to be hopeful of that. It could have been very cool, and I hope Graf Zahl will consider it for everyone's sake.

Just be careful, everyone. Dont use mean words just because their louder.

dew said:

I hoped source devs might want to explore the issue and discuss how to fix it among ports. Your question wouldn't be a Sophie's choice then. I think it's fairly obvious the willingness for that is... not universal.

Like I said multiple times already, the problem is neither to fix it or to work around it. That part is either trivial or a one-time expense. If someone can provide a reliable format check that can be integrated into all ports and all users of up-to-date engines will live happily ever after! For now, that such a check does not exist, the easiest way to deal with the problem is to toss out such blockmaps. At least it will allow to run any maps using this node builder until a better solution can be found.

No, the real problem with this new development is simply that this will fuck all over lots and lots of users of engines that CANNOT be fixed, because they are old! Why is that simple fact so hard to grasp?
This is something no port developer can do, the only way to address this issue is not to create blockmaps that do away with the long standing assumption that the first entry is always 0.

40oz said:

I dont feel that it was ignorant of you to be hopeful of that. It could have been very cool, and I hope Graf Zahl will consider it for everyone's sake.

To address the issue there needs to be some reliable detection. Missing that, the blockmap will just be recreated. Unless someone deliberately mucks this up - which actually requires some mean-spirited determination, GZDoom will play any map being processed with this node builder just fine in the future. Without demo compatibility it doesn't really matter if the resulting blockmap is a bit different.

The only way to do this is to never make a highly compressed vanilla compatible BSP. Got it. Whatever makes the ZDoom mafia happy, I suppose.

Edward850 said:

The only way to do this is to never make a highly compressed vanilla compatible BSP. Got it. Whatever makes the ZDoom mafia happy, I suppose.

Seriously Edward, are you really this dense or are you just trying to stir up a flamewar? I suppose the latter so I decline to answer because you seem to deliberately ignore what I have to say.

You never acted like this when you discovered ZDoom had the size of the spider mastermind wrong for several years. There's now some versions of ZDoom with the wrong sized spider mastermind, that somebody could end up playing.
There's a rather lot of builds of ZDoom that don't run hellfact.wad properly without rebuilding nodes. You added 3dfloors to BTSX via a compatibility text file to eliminate rendering artifacts of the GL renderer, but then there's all those older versions of GZDoom that don't have that compatibility...