Okay, interesting things happened.
First of all, yes, basic Hexen works fine in DOSbox and it's possible to create and play user-made maps. I've done it before with HCK.
Secondly, I'm no longer getting error messages from normal Hexen, it just freezes when I try to start the map. (Both command line and DOSbox.) The status window for DOSbox goes crazy with illegal read/write messages, though.
I tried source ports:
jHexen says "W_CacheLumpNum: 4348 >= numlumps" for the WAD I built with ZDBSP and "Segmentation Violation" for ZenNode.
zDOOM runs it.
Lastly, I tried opening up the Hexen IWAD, copying the real MAP01 to a separate WAD, deleting every sector and thing and replicating my 3-sector test area there. That works just fine in regular Hexen (jHexen doesn't recognize it as a PWAD, though). I'm really confused.