-------------------------------------------------------------------------------- "// Here comes the obnoxious 'visplane'."
- original Linuxdoom source release from id software -------------------------------------------------------------------------------- Let's talk about the dreaded visplane, bane of vanilla mappers everywhere.
This post assumes you understand the basics of doom mapping (linedefs, sectors, etc).   We're going to examine elements of:
1) What they are
2) How they are created
3) How some visplanes merge
4) Why some that seem like they could merge, don't
5) Potential workarounds during mapping & nodebuilding.   First off, we need to make a few clarifications about terminology:
There are two types of coordinates at work in Doom (and most other games tbh):
World-space coordinates are where stuff is in the world, X representing east-west, and Y representing north-south. Most stuff in Doom has a Z coordinate that also represents up-down, though this gets a bit complicated and is outside the scope of this particular post.
Screen-space coordinates are, well, on the screen. X is left/right, Y is up/down. The top left corner is 0,0, and these increase as you go down and right. I know this is backwards from your middle school math class, but it's just how computers work.
This matters because for example:
Imagine 2 floor sectors with the same world-space height. The world-space farther one will be screen-space above the closer one. Farther objects will be closer to the horizon. So world-space farther ceilings with the same world-space height will be screen-space below the world-space closer ones.
Here we see floors with the same height in worldspace, but the worldspace farther ones are screenspace above the worldspace closer ones.   Rows are single continuous (as in, no gaps) screen-pixel wide horizontal line segments.
Columns are single continuous screen-pixel wide vertical line segments.
Subsectors are convex pieces that make up sectors. Convex geometry is a lot easier to deal with for the engine. We'll go into some (but not all) the reasons why in a bit.
The edges of subsectors are often called "segs", we're going to call them "World Segs" to avoid confusion because there are other things in The Doom engine called segs. World segs are derived from linedefs, but sometimes the node builder splits your linedefs in half to make the subsectors.
Drawsegs are the actual edges onscreen.
A Visplane Overflow is when the vanilla/choco engine runs out of space for new visplanes (limited to 128 originally). This results in a crash.   -------------------------------------------------------------------------------- "// Now what is a visplane, anyway?"
- original Linuxdoom source release from id software -------------------------------------------------------------------------------- A visplane is a "visible plane", a region of the screen where a floor/ceiling flat is being drawn to. It's a data structure that tells the renderer where to put the flat's texture pixels on the screen.   Each visplane has 7 variables, first the 5 simple ones:
    floor/ceiling height
    flat texture
    light level
    minimum x in screen space
    maximum x in screen space   Surfaces with the same height, flat texture, and light level, can be drawn using the same calculations for what to do the texture, no matter where they are on the screen, so it makes it easier/faster to group together pixels in this manner.   The visplane is broken up into columns. So the last two variables are a pair of lists, one for the top of each column, and a corresponding one for the bottom. The columns are only defined by a screenspace top and a bottom number, and their *position* in the list is what determines where it is onscreen left-right.
There's a very important consequence of this: A visplane cannot have a screen-space vertical gap in it.
For example if we were to draw letters on the screen using visplanes: The letter U onscreen can be represented with 1 visplane, the letter O cannot.
O would require at least two visplanes.   So, the way this draws to the screen is relatively "simple":
The doom renderer goes row by row, and left-to-right in each row, and fills in the pixels that are within the boundaries of the visplane.   There are some nice videos of this in action (and more explanation) over at http://fabiensanglard.net/doomIphone/doomClassicRenderer.php   -------------------------------------------------------------------------------- Mommy, where do visplanes come from? -------------------------------------------------------------------------------- Let's step through how visplanes get made.
The nodebuilder turns our map's sectors into subsectors. Subsectors are sectors chopped into convex pieces.
So, visplanes are built from subsectors. Remember how I said visplanes can't have a vertical hole in them? This is one of the many reasons convex subsectors are easier to deal with for the engine.
But let's backtrack a little, and figure out a bit more about the order in which it visits these subsectors.   The nodebuilder splits the map in half, then splits that half into another half, and then so on, a bunch of times. This lets us be a lot faster about rendering the map for reasons that are too complicated to go into here. What's important to know is that this does happen!
This creates what's known as the bsp "tree".
The "leaves" at the end of the tree are all subsectors.   So when the renderer wants to draw the screen, it uses this BSP tree. It goes down the paths of the tree that are visible, and then once it reaches a subsector, it tries to draw it. It does this in what's called depth-first order, as in, it visits all the nodes on the front side of the tree, before going down the back side.   -------------------------------------------------------------------------------- "We've got to find that plane."
- Sean Connery as James Bond, Thunderball, 1965 -------------------------------------------------------------------------------- When each frame starts, the engine empties the list of visplanes, so all visplanes you deal with only exist in that frame. When the engine goes to draw each subsector, it runs a function called R_FindPlane twice (once for the floor, once for the ceiling). R_FindPlane looks over the list of visplanes we already have in order from oldest-to-newest, and looks for the first (oldest) visplane with the same flat texture, height, and light level. If it can't find a match, it makes a new visplane. This is the first place an overflow can happen.   Notice how it looks for an existing visplane first, meaning that "potential visplanes" can get merged in a way. It finds a "current" floor visplane, and a current ceiling visplane, and the engine keeps track of these from this step forth.   Next, the engine tries to add all the world segs to be rendered for that subsector. This is a complicated process, which we won't go into all the details of.
But it lets us define the edges of the potential visplane on screen-space.
It checks if those edges have certain properties that will change the resulting visplane. For example, if the subsectors on both sides of the world seg have the same height/light level/flat texture, it knows not to limit the visplane by them. On the other hand, if the edge corresponds to a world seg that is a 1-sided linedef wall, then it knows this will limit the visplane.
If the current world seg would limit the edge of the visplane in some way, it has to recheck the current floor/ceiling visplanes we have to make sure this won't create any of those nasty vertical holes we were talking about earlier. So it calls a function called R_CheckPlane (up to twice, once for floor and ceiling). R_CheckPlane looks over the lists of columns in the visplane already to see if they've been set to a number. If there is any overlap, R_CheckPlane adds a new visplane in the list, and makes that the current visplane. The engine just forgets about the last visplane as far as the current subsector is concerned. This is the other place where an overflow can happen.   -------------------------------------------------------------------------------- "Visplane overflows have long been a mystery (and a source of
    controversy) in the Doom editing community."
- Lee Killough, The truth about Visplane Overflows -------------------------------------------------------------------------------- So, at first glance, it seems like visplanes should nicely merge with each other if they are nearby each other, but in practice Doom seems to waste visplanes. What's the deal?   Well, it comes down to a simple problem. Doom only tries to merge with: The first & oldest visplane it encounters in the list with the same properties, and then following that, it only remembers the last one it's worked on.
Imagine the first visplane of a given type drawn on the screen is a particularly bad one, like a long horizontal strip. Then all further visplanes can't merge with each other in a good way. They only check against that first one.
Here's a simple example:

You see that vertical line splitting the floor there, despite it being seemingly possible to merge those two visplanes? They aren't merging because they only check against a subsector of the closer part below your feet. To illustrate this, let's replace the texture of the far one. Traditional wisdom would tell us this will increase the number of visplanes in the scene.

Apparently, traditional wisdom is wrong in this case.
This means that sometimes you can improve visplane counts by changing a flat, or moving a height by 1. I don't recommend you use lighting because it affects both floors and ceilings, which contribute to visplane counts separately.   While discussing this, Linguica brought forward the idea that perhaps reversing the order of the visplane search (newest-to-old) would improve merging, because newer subsectors would likely be nearer to other recent ones. I thought this was an idea I agree with, so I tried it in my choco render limits fork.
The results were definitely much better.
Here's alien vendetta map01 on default:

Here it is with the reversed order:
  Note that this method still is imperfect and results in splits which aren't ideal. I also experimented with reversing world seg order, which sometimes made things worse, and sometimes made them better. It's rather inconsistent, and I'm not sure of a definitive way to understand how yet.   While engine source code modification is beyond the scope of a vanilla mapping project unfortunately, it does give an idea of how much the merging can be improved by providing the segs/subsectors in a different order.
As we saw earlier, the nodebuilder has some control over seg order, subsector order, and just the general shape of subsectors.
This perhaps suggests we can improve visplane counts by having the nodebuilders construct subsectors to be long-parallel to hallways for example? This is still unclear at this time.   Most of this was written through discussion with Linguica,
However, further alphabetical thanks to either discussion with or resources provided by:
AlexMax, Altazimuth, Esselfortium, Fraggle, Lee Killough, Jazz Mickle, Quasar, RestlessRodent, SoM, Fabien Sanglard, and Zokum
and of course, id software.