Hexen BEHAVIOR Assembly

So the ACS page on DoomWiki says a given map (in vanilla Hexen) can only have 64 scripts defined at once. Curious, I started flipping through Hexen's source code trying to find where this is defined, among other things. (And finding out assorted other things along the way, such as the fact that the WarpTrans and Cluster fields in MAPINFO can probably go clear to 32767 without problems, since they're assigned to `short` fields and aren't used to index into arrays, AFAICT.)

 

Haven't been able to find anything about the scripts-per-map limit. But while looking through its BEHAVIOR interpreter, I was sufficiently bored, took note of the big list of opcode functions, and tried manually disassembling the BEHAVIOR lump of a map I've been working on. (Which is itself compiled with ACC, it seems.)

 

I've done some assembly-language stuff in the past, mainly with the NES (6502), so I have some background in how this sort of thing works. This is stack-based rather than register-based, which I can deal with. But the rest... 😖

 

For example, the first script I tried this with starts with the following line:

Door_Close(1, 32);

This gets compiled into a sequence of instructions like: push 1 then 32 to the stack, then call LineSpecial 10 (Door_Close) with 2 stack arguments. Okay, but there are versions of the LineSpecial functions that take number arguments directly (rather than drawing them from the stack) and it could've just used that. But this foolery is only the beginning.

 

The next line in the source script:

while (true) { Suspend; }

This is just to keep it from executing the contents again, even if it gets invoked from multiple linedefs. A simple infinite loop, and yet it gets compiled very literally: it actually pushes 1 to the stack and tests the condition every time. Any C compiler would probably optimize that into a hard infinite loop, as it should.

 

But that's not the worst of it.

 

Another script contains the following:

Delay(5*35);  // wait for 5 seconds and no I'm not gonna calculate that in my head

I figured the compiler would optimize the constant expression out. But nope, it literally carries out the multiplication and then invokes Delay with a stack argument, rather than simply invoking DelayDirect with a direct literal value of 175. 🤦🏻‍♀️

 

By this point, I'm thinking I'd rather write my scripts in assembly rather than use ACC.

 

Because it might be more expressive than ACS language itself. For example, the way it assembles strings for printing is to refer to them by index... and it pulls these indexes from the stack rather than using a direct value. The only benefit of going through the stack is to allow for values to be computed at runtime... which, in this case, means they can be. That means you could lookup strings by a runtime-generated index, or manually iterate through a list, rather than having to specify each one in a hard-coded expression. ACS language doesn't have a way to do that, that I know of.

 

For example, I tried my hand at writing an assembly version of a function to the effect of "print each of N strings one at a time with a Suspend call between each" (so that each time the script is called, it prints the next one). In ACS language, all you can do is something like:

script 123 (void) {
    print(s:"ONCE UPON A MIDNIGHT DREARY"); Suspend;
    print(s:"WHILE I PONDERED, WEAK AND WEARY"); Suspend;
    print(s:"OVER MANY A QUAINT AND CURIOUS"); Suspend;
    print(s:"VOLUME OF FORGOTTEN LORE");
}

It'll probably compile this exactly as written: set up a string, print it, call Suspend, set up the next string, print that, call Suspend again, and so on. But if you write BEHAVIOR instructions directly, you could do that sort of thing in a loop.

; Assumption: that the strings are indexed consecutively.
; FIRST is the index of the first string in the list.
; AFTER is the index just after the last string in the list.

	Push FIRST
	AssignScriptVar 0	; ScriptVar 0 <-- pop from stack
	Push FIRST		; keep a copy there
Loop:
	BeginPrint		; initializes print buffer
	PrintString		; pops string index from stack
	EndPrint		; displays message to player
	IncScriptVar 0		; counter++
	PushScriptVar 0		; copy to stack...
	CaseGoto AFTER, Done	; if counter == AFTER, done
	Suspend			; will resume here next time
	Goto Loop
Done:
	; whatever comes next, or simply Terminate.

You can find the definition of each in p_acs.c, in "Cmd"+name, such as CmdCaseGoto at line 1531.

 

CaseGoto is used for switch/case constructs, I assume; if the provided value equals the value at the top of the stack, then it jumps to the given location and pops the value off the stack, otherwise it leaves it there and continues, presumably for another CaseGoto statement. So it'll clear the copy when Done, but leaves it there if we're going on to the next string, and there it will remain for the next PrintString. (Which is also why I push an extra copy to the stack before the Loop.)

 

And since you're doing all this manually, you can assign strings to indexes manually, so you can ensure it works as expected. And since you know the string indexes for everything, you could use them as arguments in Script Execute line specials in the map and just feed that to PrintString rather than using a big fat switch/case statement that produces big fat code with big fat 32-bit integers for all these byte-sized values.

 

So anyhow, are there any assembly-type BEHAVIOR compilers out there? Or a better version of ACC or something (that's compatible with vanilla Hexen)? At this point I almost wanna try my hand at writing a BEHAVIOR assembler in Ruby and see if I can trick Doom Builder into using that instead of ACC. (Or instead of, you know, actually working on my map. 🙍🏻‍♀️)

Edited February 20, 2020 by Rainne

take a look at GDCC. i vaguely remember that it had "vanilla-compatible" mode.

 

4 hours ago, Rainne said:

So anyhow, are there any assembly-type BEHAVIOR compilers out there? Or a better version of ACC or something (that's compatible with vanilla Hexen)? At this point I almost wanna try my hand at writing a BEHAVIOR assembler in Ruby and see if I can trick Doom Builder into using that instead of ACC. (Or instead of, you know, actually working on my map. 🙍🏻‍♀️)

 

Why does this inefficient code bother you?

It worked fine in 1995 when computers were a lot slower than today. The speed hit by these things is mostly insignificant, unless you run very complex scripts on very complex maps, neither of which makes any sense in vanilla Hexen.

 

In general you have to assume that ACC is dumb as shit - it can't do anything more than a straight translation of your input into bytecode output - it cannot even do 'if' expression shortcuts.

For example, those variants of certain functions that take direct parameters only get used if you prefix the first one with 'const:'.

 

GDCC is to be taken with care, it may be a more efficient compiler, but it's also a lot more complex, meaning more chances of things going wrong because it often has to work around the byte code's deficiencies and creates even less optimal code.

 

Regarding ACS assembly - no, such a thing does not exist. Nobody needs it and you'd have to look out for the interpreter's quirks yourself instead of having a compiler doing it. But as I said, the compiler takes no room for interpretation, it literally translates your imnput into output, so it's a lot closer to assembly language than a modern optimizing C compiler.

1 hour ago, Graf Zahl said:

GDCC is to be taken with care, it may be a more efficient compiler, but it's also a lot more complex, meaning more chances of things going wrong because it often has to work around the byte code's deficiencies and creates even less optimal code.

GDCC has "ACS mode" (at least it had last time i checked), where it doesn't really try to be compatible with C, but still utilise it's optimising codegen, tho. also, GDCC has "as" too, but i never ever looked at it.

Considering the issues we had in the past with GDCC generated code I'd still recommend against it, optimizing CC and all.

I skimmed over GDCC, and its ACC seems oriented around ZDoom's NuACS rather than anything Hexen knows. Even so, I'd rather use an assembler. Doing stupid stuff like this in a hex editor is tedious. 🙍🏻‍♀️

 

Pick any class and difficulty and see what everyone's favorite Serpent-turned-Rider has to say!

misbehavior.zip

Writing assembler is tedious and error-prone, you're gonna be the only person to ever want to do it (which doesn't mean you shouldn't create an assembler -- if that's how you wanna spend some time then fair enough I think).

i am pretty sure that i've seen some very old ACS decompiler that had ACS assembler. but don't even ask me how it was titled. alas. i mean, OP won't be the only person, there were people who thought that it will be fun idea to implement. ;-)

Note that with an assembler you forfeit any sanity checks the compiler may provide, subject the VM to crashes and aborts and won't gain anything of value.

If you got a script that's simple enough to be done in ACS assembly you'll never EVER experience any performance issues or achieve significant gains in code size. All you achieve is a lot of wasted time. The performance characteristics of the bytecode interpreter are mostly completely irrelevant because the bulk of time is not being spent interpreting code but running the complex functions that can be called.

 

ACS is also a pure stack VM, it got no registers or other niceties to allow writing assembly code efficiently. Even doing something seemingly simple as a 'for' loop can cause you serious trouble to code.

 

Graf, i believe you're getting it slightly wrong. i don't think that OP is concerned with speed, it is more of a hobby. that kind of hobby that makes people write games in assembler for retro-machines, and such.

 

p.s.: also, i'm still toying with the idea of writing a Forth compiler for ACS VM. another useless, but very fun (at least for the author) thing. sadly, i need two stacks for Forth.

19 hours ago, Graf Zahl said:

Note that with an assembler you [...] won't gain anything of value.

You gain the ability to select strings at runtime within expressions. For example, an ambient-sound loop could select first_sound_index + Random(0,5) to choose a sound to play, rather than using a select/case loop. Or play Mad Libs with strings, like in the example WAD I posted. I don't know of any way to declare strings up-front (rather than as string literal expressions within a script) and know their indexes and ensure they're contiguous so you can do stuff like AmbientSound(sounds+Random(0,5), 80) or iterate over them -- similar to what you can do with sector tags -- but this is (relatively) straightforward when assembling the BEHAVIOR lump manually.

 

And it's not as difficult as you need it to be. I do have some assembly-language background (6502 machine code for NES ROM hacking, and the in-game programmable chips in Stationeers use a MIPS-like syntax, from what I'm told). I even wrote a simple disassembler in Ruby and am looking over the code from some of the stock maps, like Winnowing Hall and Darkmere, and getting a feel for how things are done. I could totally do this.

 

And at this rate, with all the boilerplate I'd have to add to ACC language to make it do what I want, it'd be easier to write it directly in assembly.

• Having to add "const:" to non-computed expressions. But not the random builtins that don't have a 'direct' version or you get a compiler error for your trouble and have to un-add it. You just have to memorize which ones you're allowed to use it with and which ones you're not.
• "Delay(wait * Random(const:3,6));" gives me "Tried to POP empty expression stack" 🤪
• "Delay(Random(const:3,6) * wait);" gives me "Syntax error in constant expression" (but it works fine by itself, like "switch (Random(const:1,3))")
• I can tell you exactly what instruction sequence those should compile to, down to the byte if I can figure out which Notepad window has the opcode list I drew up.

🤷🏻‍♀️

 

also, is there such thing as ACS opcodes techspecs?

I got that from the source code (p_acs.c); following how it reads the BEHAVIOR lump to infer its layout. It doesn't have a table explicitly mapping opcodes to functions, but does so implicitly with a big array of function pointers; by reading the functions themselves, one can infer what parameters it takes and where it gets them (direct arguments vs stack) and what it does with them.

yeah, i guess everybody used the source as the reference. and nobody bothered to write a specs. ;-) of course, i'm not blaming people for not doing it, after all i didn't wrote it too for extended ZDoom opcodes. ;-) i just hoped that maybe you found such document. ;-)

51 minutes ago, Rainne said:

You gain the ability to select strings at runtime within expressions. For example, an ambient-sound loop could select first_sound_index + Random(0,5) to choose a sound to play, rather than using a select/case loop. Or play Mad Libs with strings, like in the example WAD I posted. I don't know of any way to declare strings up-front (rather than as string literal expressions within a script) and know their indexes and ensure they're contiguous so you can do stuff like AmbientSound(sounds+Random(0,5), 80) or iterate over them -- similar to what you can do with sector tags -- but this is (relatively) straightforward when assembling the BEHAVIOR lump manually.

 

I strongly, strongly recommend agains abusing string indices. This is very likely to break with ACS engines that recompile the code or reorganize the data internally. You are in severe danger of creating undefined behavior with such an approach - convenient as it may look.

ZDoom's ACS VM performs some code analysis to guess the used string indices, and one assumption it makes is that string indices in the byte code are constants, not expressions so you may easily run afoul of this.

 

Don't also forget: Currently there's three main ACS VMs out there - Hexen's original, ZDoom's and Eternity's - they are all very different and you got zero guarantee that string indices are being handled compatibly.

 

 

 

51 minutes ago, Rainne said:

 

And at this rate, with all the boilerplate I'd have to add to ACC language to make it do what I want, it'd be easier to write it directly in assembly.

• Having to add "const:" to non-computed expressions. But not the random builtins that don't have a 'direct' version or you get a compiler error for your trouble and have to un-add it. You just have to memorize which ones you're allowed to use it with and which ones you're not.

 

Then don't add "const:"! Here's a little secret: It doesn't reduce code size much and certainly does not improve performance significantly. There's a reason why ZDoom has stopped adding const versions of new features early in its history.

 

 

51 minutes ago, Rainne said:

• "Delay(wait * Random(const:3,6));" gives me "Tried to POP empty expression stack" 🤪

 

Again: Do not use "const:" ! In particular, do not use const in non-constant contexts!

 

51 minutes ago, Rainne said:

• "Delay(Random(const:3,6) * wait);" gives me "Syntax error in constant expression" (but it works fine by itself, like "switch (Random(const:1,3))")

 

Do I need to repeat myself: Do not use "const:"!

 

51 minutes ago, Rainne said:

• I can tell you exactly what instruction sequence those should compile to, down to the byte if I can figure out which Notepad window has the opcode list I drew up.

🤷🏻‍♀️

 

 

Here's another hint: If you are dead set on this route, there will be zero support if the map does not work.

I still don't understand what you try to gain here. It is very easy to screw up the ACS stack by writing badly constructed code. Such bad code may work in Hexen, but cause an abort in ZDoom - or maybe not even in ZDoom but Eternity or vice versa. These VMs are all different and may throw up on different error conditions. And who knows, it may appear to work but somehow get bad values somewhere in a non-obvious way and run off the rails on these values.

 

And should that happen my only response will be "Please complain to the mapper who wouldn't listen to common sense".

 

39 minutes ago, Graf Zahl said:

Currently there's three main ACS VMs out there

four. ;-)

Have you written another one? Vavoom's was based on ZDoom's.

 

yep. but GZDoom VM evolved since (the fork was over a decade ago after all), and i am slowly changing mine too. it is not radically different, yet it has its own ways to do some things (and will prolly change even more in the future). so it can be considered the 4th one, i guess.

3 hours ago, Graf Zahl said:

This is very likely to break with ACS engines that recompile the code or reorganize the data internally. You are in severe danger of creating undefined behavior with such an approach - convenient as it may look.

ZDoom's ACS VM performs some code analysis to guess the used string indices, and one assumption it makes is that string indices in the byte code are constants, not expressions so you may easily run afoul of this.

Except my code is well-defined. I'm not using undefined opcodes or self-modifying code or out-of-bounds array indexes here (nor do I want to); simply using an instruction in the way the original interpreter expects it to be used: string indexes, obtained from the stack. ACC is limited to only providing fixed values for these indexes with little control over which strings get which index, but that's an ACC limitation, not a BEHAVIOR limitation. I have no intention of indexing beyond the end of the string array (which would indeed be undefined behavior and the responsibility would be mine alone) or otherwise feeding PrintString anything that's not a valid string index; merely selecting among valid strings at runtime.

 

As has been pointed out, there isn't any specification out there (presumably the original devs had something of the sort, because, after all, Hexen and ACC need to agree on how BEHAVIOR works). In the absence of one, Hexen's VM source code is the closest thing we have to a specification, and everything I've done with BEHAVIOR so far has been in accordance with it.

 

And as you have pointed out, the original VM already performed perfectly well, even on 1995 hardware. I don't see what value there is in recompiling it (even to native machine code, let alone a different soft bytecode format) -- it's likely to cause problems without even solving any, merely becoming more fragile. 💁🏻‍♀️ Maybe as a fun hobby project, sure -- that's kinda what this thing is -- but it's still charged with executing BEHAVIOR code, and if its assumptions prevent it from doing so properly, then its assumptions are flawed. And it alone is responsible for such flaws: unless you think ZDoom has more authority to define what Hexen is than Hexen itself does? In which case it warrants my original reply that got swept under the rug. 🙄

 

Because I'm mapping for Hexen (and properly: not relying on any undefined, non-Hexen behavior), and your port is only relevant to my map to the extent it implements Hexen. From what you say, it may have limited ability to do that, which therefore puts an asterisk on its claim to even be a port of Hexen. I'd like my map to work with both Hexen and ZDoom; but if they disagree about my map, and the disagreement can't be reconciled, Hexen wins. After all, your port may deviate from Hexen in some direction, while another port may deviate in another direction, but they all orbit around Hexen itself. Moons do not define a planet; if the moon strays from its orbit, it is the moon's status that changes, not the planet's. And your port's orbit seems to have a little wobble to it.

 

And if this all comes down to assumptions about what is and isn't constant? Well, I'd like to help with that by tagging such parts of my code! After all, it doesn't just produce less-tedious bytecode, but also gives hints about semantics that a reinterpreter could take advantage of. But you keep telling me not to do that. 🤷🏻‍♀️ For example, if your reinterpreter comes across Random(const:0,255), it knows it can depend on those numbers and could translate it into an unadorned call to P_Random(), rather than having to add supporting arithmetic and range-checking/error-handling to handle unknown inputs. But if it's a plain, from-the-stack Random, it can't make such assumptions (without some separate validation like being preceded only by PushNumber calls), or it's likely to fail at the very job it signed up for: implementing BEHAVIOR.

A lot of the Raven support code in ZDoom is older than the release of the source code. That includes the ACS VM. It was originally created by reverse-engineering the bytecode format and doing something that can run it. The fact that it is an independent implementation can potentially cause slight differences in edge cases, and if you're not using the ACC tool released by Raven Software, then you are relying on undefined behavior.

28 minutes ago, Rainne said:

Except my code is well-defined. I'm not using undefined opcodes or self-modifying code or out-of-bounds array indexes here (nor do I want to); simply using an instruction in the way the original interpreter expects it to be used: string indexes, obtained from the stack. ACC is limited to only providing fixed values for these indexes with little control over which strings get which index, but that's an ACC limitation, not a BEHAVIOR limitation. I have no intention of indexing beyond the end of the string array (which would indeed be undefined behavior and the responsibility would be mine alone) or otherwise feeding PrintString anything that's not a valid string index; merely selecting among valid strings at runtime.

 

As has been pointed out, there isn't any specification out there (presumably the original devs had something of the sort, because, after all, Hexen and ACC need to agree on how BEHAVIOR works). In the absence of one, Hexen's VM source code is the closest thing we have to a specification, and everything I've done with BEHAVIOR so far has been in accordance with it.

 

I was just pointing out where an altered VM may fail if it works through the code by taking some known invariants from ACC output for granted.It doesn't necessarily have to fail but it cannot be ruled out.

 

 

28 minutes ago, Rainne said:

 

And as you have pointed out, the original VM already performed perfectly well, even on 1995 hardware. I don't see what value there is in recompiling it (even to native machine code, let alone a different soft bytecode format) -- it's likely to cause problems without even solving any, merely becoming more fragile. 💁🏻‍♀️ Maybe as a fun hobby project, sure -- that's kinda what this thing is -- but it's still charged with executing BEHAVIOR code, and if its assumptions prevent it from doing so properly, then its assumptions are flawed. And it alone is responsible for such flaws: unless you think ZDoom has more authority to define what Hexen is than Hexen itself does? In which case it warrants my original reply that got swept under the rug. 🙄

 

You have to ask Team Eternity here. Apparently someone over there thought that executing ACS code directly is a hazard for stability or whatever else. ZDoom never did it but the VM exists and I have zero clue how it goes about translating the original byte code. If it makes assumptions about code layout it may just fail to handle code that has an unusual pattern. We'd need input from someone with better knowledge to explain how this works.

 

 

 

28 minutes ago, Rainne said:

Because I'm mapping for Hexen (and properly: not relying on any undefined, non-Hexen behavior), and your port is only relevant to my map to the extent it implements Hexen. From what you say, it may have limited ability to do that, which therefore puts an asterisk on its claim to even be a port of Hexen. I'd like my map to work with both Hexen and ZDoom; but if they disagree about my map, and the disagreement can't be reconciled, Hexen wins. After all, your port may deviate from Hexen in some direction, while another port may deviate in another direction, but they all orbit around Hexen itself. Moons do not define a planet; if the moon strays from its orbit, it is the moon's status that changes, not the planet's. And your port's orbit seems to have a little wobble to it.

 

Right now the ZDoom family of ports, Doomsday and Vavoom are the only advanced ports supporting Hexen and the only one still using the original VM here is Doomsday, which is not a port being used by players of custom mods. So you are essentially targeting a vacuum here if you consider vanilla Hexen as the only reference. Most of the time it will work but there's no guarantees. Should anyone report your mod for being bugged and the only source I can find is ACS assembly the bug with closed right away.

 

 

28 minutes ago, Rainne said:

 

And if this all comes down to assumptions about what is and isn't constant? Well, I'd like to help with that by tagging such parts of my code! After all, it doesn't just produce less-tedious bytecode, but also gives hints about semantics that a reinterpreter could take advantage of. But you keep telling me not to do that. 🤷🏻‍♀️ For example, if your reinterpreter comes across Random(const:0,255), it knows it can depend on those numbers and could translate it into an unadorned call to P_Random(), rather than having to add supporting arithmetic and range-checking/error-handling to handle unknown inputs. But if it's a plain, from-the-stack Random, it can't make such assumptions (without some separate validation like being preceded only by PushNumber calls), or it's likely to fail at the very job it signed up for: implementing BEHAVIOR.

 

const: was just a cheap-ass way to make the code shorter. It wasn't made to make the code faster. Ultimately it even failed in making the code shorter because the engine side support code for the added byte codes costs more space than it saves in the script data. And if some const: constructs do not work, it's solely the fault of ACC, not the VM.

And with ZDoom's packed code it doesn't even make the code shorter anymore.

 

23 minutes ago, Gez said:

A lot of the Raven support code in ZDoom is older than the release of the source code. That includes the ACS VM. It was originally created by reverse-engineering the bytecode format and doing something that can run it. The fact that it is an independent implementation can potentially cause slight differences in edge cases, and if you're not using the ACC tool released by Raven Software, then you are relying on undefined behavior.

No, what I'm relying on is the source code that, for lack of anything that can better claim to do so, defines it. That code is available to ZDoom if they need to update their implementation where it produces different results than the original for lack of knowledge when it was written. It's ZDoom's responsibility to conform to Hexen, not the other way around.

 

If there was ever an official specification of the BEHAVIOR bytecode, it was never released; by contrast, this was released for mappers to refer to. The ACS section concerns itself mainly with ACC language rather than the bytecode it compiles to. So the only official thing it has to say about what I'm playing with:

Quote

There is only one data type ACS, a 4 byte integer. Use the keyword int to declare an integer variable. You may also use the keyword str, it is synonymous with int. It's used to indicate that you'll be using the variable as a string. The compiler doesn't use string pointers, it uses string handles, which are just integers.

This doesn't necessarily support what I'm doing, but it doesn't contradict it either. And when designing the BEHAVIOR foundation that's meant to implement this, they saw fit to have the PrintString instruction take a stack argument (which is variable) rather than a direct argument (constant); in fact it doesn't even have a version that takes a direct argument. And you can even take advantage of this in a completely clean way:

str whatever;

script 1 (int arg) {
    switch (arg) {
    	case 1: whatever = "HELLO"; break;
        case 2: whatever = "HOWDY"; break;
        default: whatever = "GOODBYE";
    }
    print(s:whatever);
}

What argument is there that this is invalid code? And yet the only way to perform the print is to refer to the variable (PushMapVar then PrintString). QED.

2 minutes ago, Rainne said:

If there was ever an official specification of the BEHAVIOR bytecode, it was never released; by contrast, this was released for mappers to refer to. The ACS section concerns itself mainly with ACC language rather than the bytecode it compiles to.

Yes. That means that the official specificiations are for the ACS language as compiled by ACC, and that anything that shortcuts it is undefined behavior.

On 2/20/2020 at 5:55 AM, Rainne said:

It'll probably compile this exactly as written: set up a string, print it, call Suspend, set up the next string, print that, call Suspend again, and so on. But if you write BEHAVIOR instructions directly, you could do that sort of thing in a loop.

...and that first sample of yours is wrong. because `Suspend` doesn't carry on stack contents. actually, it expects empty stack. i don't know if the original code resets SP each time a script is awaken, but the script is still wrong in any case. if there's no stack reset, then the things will break horribly if you have two such "non-clean" scripts. and if there is, welcome to UB.

 

doing tricks is fun, but doing tricks without proper specs is... not so fun. ;-)

Weirdly enough I was just thinking about ACS and the bytecode and how it hasn't received much attention. So, fantastic  @Rainne  that you are looking at this and I wish you great success!