Scripting - The main challenge for Doom's future

I'm here to talk about scripting, and why it remains such a problem for Doom ports in general, what the specific needs for Doom scripting are, and how they could be solved.

Scripting is widely recognized as an absolute necessity in modern game engines, where the programmers' time is not best spent attempting to anticipate every need of the map designers and reorganizing internal workings for level-specific hacks, as was a frequent practice in the old days.

Doom has been plagued since day one with the search for a suitable scripting language - one that would be capable of not just providing a toy interface to a small set of line specials and built-in functions, but one that could be used to actually replace parts of the native game code. We have seen many attempts, including ACS, FraggleScript, RTS, et al. But none have reached this goal so far, and all have been developed or maintained in isolation with no goal of generating reusable components.

In order to achieve this goal, the language and its runtime environment would need at minimum to meet the following requirements:

1. Its representation should be textual, and therefore widely portable. Compilation should be done at load time, or just-in-time. Including bytecode in WAD files should not be required. This in turn requires the compiler to be relatively compact and self-contained, so that it may be easily embedded in the host program.

2. It should be capable of compiling code fragments, allowing a "code-anywhere" approach. This would enable embedding of scripts inside other established input languages (ex: EDF). The compiler and virtual machine must both be capable of functioning on data buffers, and not be built to specifically use disk files or certain types of archives (wads, zips, etc.).

3. It should have a simple and efficient binding to the C language. Calling a C function from the scripting language should involve the provision of a name/function-pointer pair to the interpreter. Calling a script from C should involve passing a name or numeric index along with argument data to the interpreter. It should not involve low-level bothers such as stack manipulation through dozens of function calls, which is both difficult to use and extremely slow.

4. The virtual machine must be capable of saving and restoring state, both by sleeping scripts and by providing access to all internal registers and memory segments for purposes of serialization to disk file.

These four basic tenets of functionality are amazingly met by virtually no extant scripting languages, Doom-specific or otherwise. ACS fails two out of four. Lua fails two out of four. Small fails three out of four.

A large majority of modern scripting languages have evolved into application-domain languages, meaning they are no longer suitable for embedding at all. Their compilers are huge, they require installation of a runtime package on the user's system in order to function, and their standard libraries are incapable of being sandboxed in order to prevent users from doing damage to the system (malicious wad files, anyone?).

On the other end, we have "garage" languages designed by hobbyists because they felt like testing their mettle at language design. These are typically easily embedded (like Small), but are plagued by deep design issues and severe compiler bugs. The lack of people actually using these languages means that they have scant been tested. Stability is a foremost requirement, so these languages fail to contend.

From where I stand, I cannot see a solution to this issue. Even if we as a community were to work together, there is little guarantee that would produce a language meeting these criteria which would be well-designed and stable. Compiler writing is one of the more difficult tasks in programming, typically tackled only in graduate classes and even then at a largely useless theoretical level - finding actual practical, reusable ideas for implementation without simply examining the source of an already-existing compiler is almost impossible.

I myself have done extensive research, and my best effort to date has only produced a very basic recursive descent parser for expression evaluation. My knowledge of the process of code generation is nearly non-existent, and remains the primary roadblock for the Eternity Engine.

I'd like to see ideas getting thrown around just the same, however. I think we all stand to benefit from an open process of discussion on the matter, rather than a lot of independent efforts all trying to do the same thing behind closed doors.

tl;dr version : Doom needs a specific scripting language because the rest aren't really great options.

Let's not forget one other fundamental issue that's possibly prohibitive towards a common language for multiple ports:

Much of what gets exposed depends on the port's internal implementation. If you try to do something for a port that still uses the original mobjinfo table and one that uses true class hierarchies you won't ever be able to get to a point that will make both ports' authors happy.

I'm not really concerned with the idea of a unified API for Doom -- that's obviously never going to be possible. Even two very similar ports based off each other will have differing needs with respect to this sort of thing. What I'm curious about is whether the rest of the workings could have some commonality - parsing, evaluation, code generation, and interpretation. At any rate I am open to working with any party toward such ends myself.

You know ZDoom's catchphrase, 'WFDS' right?

That said, I already started working on a compiler/interpreter *twice* and got shot down by the big master each time - so I'm sorry I won't put any more work into it - because I know where it will end.

Quasar said:

3. It should have a simple and efficient binding to the C language. Calling a C function from the scripting language should involve the provision of a name/function-pointer pair to the interpreter. Calling a script from C should involve passing a name or numeric index along with argument data to the interpreter. It should not involve low-level bothers such as stack manipulation through dozens of function calls, which is both difficult to use and extremely slow.

I assume you want something like what FraggleScript did, which just had a table of function names with callback functions to invoke for each of them and a "argv"-style system for parsing arguments? It seems to me that you could probably write wrappers for almost any language to do something like that. Ruby's C interface is already a lot like that, Python appears to let you do the same thing. Lua's system is a bit more convoluted but it's not that hard to see how you could implement what you describe on top of what it gives you.

4. The virtual machine must be capable of saving and restoring state, both by sleeping scripts and by providing access to all internal registers and memory segments for purposes of serialization to disk file.

This is probably the biggest problem. You can do this two ways, really -

• the first, and easier to implement, way is to dump the responsibility on the people writing the script - you make anyone writing a script implement "load" and "save" functions, and if they don't, their save games don't work. For obvious reasons, this is not preferable.
  
• the more difficult to implement option involves automatically serializing the contents of the execution environment such that you can reload it back and continue exactly where you left off after loading a savegame.
  
  ACS has an advantage on this front because (1) the language is simple and (2) the interpreter is self-contained and designed with this requirement in mind, so it's (comparatively) easy to serialize.
  
  Nonetheless, I don't think we should overestimate the difficulty of doing this. Most languages include support for this kind of thing, and although it may be a bit tricky, with a bit of brain-power, it should be possible to work out.
  

In summary, I think something to realise is - scripting for games is a particular niche use of scripting languages in general. You'll probably find that none of the languages you investigate will do exactly what you want. But if you examine things carefully, you'll probably find that you can get what you want or at least almost what you want with a bit of work. It can only be less work than designing and implementing your own language, after all!

tl;dr version : Doom needs a specific scripting language because the rest aren't really great options.

That isn't what this thread is about at all. If you don't understand what's being talked about, how about you just not post instead?

fraggle said:

In summary, I think something to realise is - scripting for games is a particular niche use of scripting languages in general. You'll probably find that none of the languages you investigate will do exactly what you want. But if you examine things carefully, you'll probably find that you can get what you want or at least almost what you want with a bit of work. It can only be less work than designing and implementing your own language, after all!

Well, most scripting languages I investigated didn't even come close to the feature set I'd like to have for ZDoom or I would have used one already. The area where most languages are insufficient is with class hierarchy features - something that's essential to expose ZDoom's internal features. Without any simple way to get to the internal classes and data structures (and being able to inherit from them) the languages have already failed to meed the mminimum requirement I set so I fear that a specific implementation is inevitable.

Graf Zahl said:

Well, most scripting languages I investigated didn't even come close to the feature set I'd like to have for ZDoom or I would have used one already. The area where most languages are insufficient is with class hierarchy features - something that's essential to expose ZDoom's internal features. Without any simple way to get to the internal classes and data structures (and being able to inherit from them) the languages have already failed to meed the mminimum requirement I set so I fear that a specific implementation is inevitable.

You mean like the ability to access members of objects and get/set their values? I know that Python has struct objects which may be able to do something like that. For Chocolate Doom's dehacked parser I wrote some slightly evil code to map out C structures programatically to allow fields to be set; presumably something similar could be done for a scripting language to map out internal structures?

Graf Zahl said:

You know ZDoom's catchphrase, 'WFDS' right?

That said, I already started working on a compiler/interpreter *twice* and got shot down by the big master each time - so I'm sorry I won't put any more work into it - because I know where it will end.

Sorry to hear, I can understand the frustration that would lead to.

fraggle said:

I assume you want something like what FraggleScript did, which just had a table of function names with callback functions to invoke for each of them and a "argv"-style system for parsing arguments? It seems to me that you could probably write wrappers for almost any language to do something like that. Ruby's C interface is already a lot like that, Python appears to let you do the same thing. Lua's system is a bit more convoluted but it's not that hard to see how you could implement what you describe on top of what it gives you.

Yes, FraggleScript was right on in this respect. Small also works this way. Calling a Small function, or calling Small functions from C, is dead simple.

I do realize that it could probably be layered on top of Lua's stack system. It still remains to be seen if Lua has reasonable support for serialization, however. The only thing I can find is a third party library which only achieves this by apparently reaching under the hood. It is called Pluto and was evidently only recently completed in the last couple of months.

However, whether or not *I* can achieve that layering is questionable. The entire stack API rather makes my head spin in circles.

fraggle said:

* the first, and easier to implement, way is to dump the responsibility on the people writing the script - you make anyone writing a script implement "load" and "save" functions, and if they don't, their save games don't work. For obvious reasons, this is not preferable.
* the more difficult to implement option involves automatically serializing the contents of the execution environment such that you can reload it back and continue exactly where you left off after loading a savegame.

ACS has an advantage on this front because (1) the language is simple and (2) the interpreter is self-contained and designed with this requirement in mind, so it's (comparatively) easy to serialize.

Nonetheless, I don't think we should overestimate the difficulty of doing this. Most languages include support for this kind of thing, and although it may be a bit tricky, with a bit of brain-power, it should be possible to work out.

The difficulty depends on the language in question. Small for example, which cannot interrupt a script during its execution, only requires the data segment to be saved, and its interpreter has a built-in function for this purpose. Some languages' VM's operate too closely to the hardware however, and do not have contiguous data segments to hand you - JavaScript for example uses a full-blown garbage collected heap. Serializing that sucker wouldn't be happening any time soon, and would lead to a 40 MB game save file in any case. For a language like JS, user-based serialization would be the only possibility. It is something that I do not feel most Doom mod authors would be willing to accept, since the bar has been set by languages like ACS such that automation is expected.

fraggle said:

In summary, I think something to realise is - scripting for games is a particular niche use of scripting languages in general. You'll probably find that none of the languages you investigate will do exactly what you want. But if you examine things carefully, you'll probably find that you can get what you want or at least almost what you want with a bit of work. It can only be less work than designing and implementing your own language, after all!

I keep looking but I keep coming away with the feeling that it's never going to work. Lua, for example, is missing a lot of things that you'd want in order to move native code out into it. You could do it, but you just end up making it very slow and much larger, losing the essence of the logic in repeated calls to native functions just to, for example, test a flag value (this was discussed previously in the EE forums). I'd like to be pleasantly surprised, however. That's one thing this thread is about - maybe somebody knows more about some of these languages than I've been able to ascertain just by looking over their webpages and docs ;)

I've been using Lua for a long time with OBLIGE, and more recently have used it for the HUD drawing system in EDGE.

As a general purpose scripting language in a DOOM engine, I think Lua could be very nice. However there are some issues that stop me from using it fully:

Number #1 issue is serializing of state for savegames. That is really hard. For example, how to know when some table is part of the environment and should not be saved (e.g. the "math" library). Debugging any problems in the serialization code would be a nightmare. Plus I can't shift the burden to script writers to implement load/save functions, that is way too much to expect of modders.

Number #2 issue is speed and memory usage, I fear that 1000 monsters each running a little bit of Lua code every few tics is going to slow everything to a crawl. Now I gotta say Lua is pretty fast for an interpreted language, in OBLIGE it can process large maps (converting it from block/fragment representation to lines/sectors/etc) in a few seconds on a 500Mhz machine. However a game engine is a different beast and halting the gameplay for even 1/5th second to perform garbage collection is not acceptable.

Number #3 issue: the ability of a script to go into an infinite loop and halt the whole game engine. Some modification to the VM would be needed to prevent this.

Number #4 issue: the Lua syntax uses "end" keyword to end every type of block (if,while,for,function). This is a real headache when there is a unclosed block somewhere, it can take a while to find where the problem is. That could get really frustrating for modders.

Number #5: not a good fit with the engine's internal C++ classes. Now for me the scripting language should not be so tightly coupled that it could, for example, directly modify a mobj_t's "x" field (and hence mess up the blockmap linkage), instead I'd provide a move_thing() function. However for read access it gets a bit painful to write a whole lot of get_xxx() functions. Well I guess this issue (and the question of what engine state can be moved into the Lua code) applies to any scripting language, but a more OO scripting language could potentially have a much nicer interface with the engine.

Some of these issues are not major, you could live with bad syntax or potential for infinite loops. Serialization and speed are the main ones for me.

I would say that item 1 in your list is probably the least necessary of the four - working off of bytecode, instead of using an interpreter or JIT compiler, is a relatively minor hassle. (It's still nice to have the original source, obviously, both for "how did they do that" and "well, this almost covers what I want... maybe a few little tweaks..." purposes.)

That said: while I'd love to try my hand at coming up with a superset of ACS and RTS (and possibly FS) as far as functionality goes, that's probably better left to someone with a better track record of finishing things.

3. It should have a simple and efficient binding to the C language ...... It should not involve low-level bothers such as stack manipulation through dozens of function calls, which is both difficult to use and extremely slow.

Well I personally don't find Lua's stack-based interface difficult to use, and I can't see it being slower than any other method of passing data between Lua functions and native code. If you had ever programmed with a stack-based language like FORTH, or even in pure assembly, then you'd probably understand it a lot better.

For example when a C function call is compiled into assembler: each argument is pushed onto the runtime stack, the function is called, and when the function returns the values are popped off the stack. [This is not the only way to do it, but on the register-starved x86 that is the most common way]. So Lua's function call API actually reflects how C function calls work.

While I'm not sure that Lua could be a suitable scripting language for DOOM ports, even if heavily tweaked to overcome the shortcomings described here, I also doubt that beefing up Small (aka Pawn) to have enough functionality is feasible. There are people here with the skill to create something new, but not the will.

MDenham said:

I would say that item 1 in your list is probably the least necessary of the four - working off of bytecode, instead of using an interpreter or JIT compiler, is a relatively minor hassle. (It's still nice to have the original source, obviously, both for "how did they do that" and "well, this almost covers what I want... maybe a few little tweaks..." purposes.)
finishing things.

The main reason that I find it a necessity is precisely because Small has a severe problem with this. If its compiler is run on a 64-bit target, it will generate 64-bit bytecode. There is no metadata within the bytecode unit to tell whether the file is 32- or 64-bit. This makes Small bytecode non-portable. I asked the author to fully repair this and he declined. Small was targeted primarily toward embedded systems (it has been used in the iPod firmware, for example), and those are always specific to a certain processor, so it's not an important goal to him I guess.

Ajapted said:

For example when a C function call is compiled into assembler: each argument is pushed onto the runtime stack, the function is called, and when the function returns the values are popped off the stack. [This is not the only way to do it, but on the register-starved x86 that is the most common way]. So Lua's function call API actually reflects how C function calls work.

Yep except to push arguments on the system stack is a single instruction. To push arguments onto the Lua stack is a function call, which in C, performs system stack pushes before it ever gets to modify the Lua stack. The native instruction to VM instruction ratio is really bad with this sort of approach, even if I can see the logic in trying to shield the client code from the internal implementation.

Right now, fraggle has me looking at Python again. I have to say I'm starting to be won over. This is mainly contingent on how well it's going to work on systems where it's not installed, and that I will be finding out very soon with a test program :)

Quasar said:

The main reason that I find it a necessity is precisely because Small has a severe problem with this. If its compiler is run on a 64-bit target, it will generate 64-bit bytecode. There is no metadata within the bytecode unit to tell whether the file is 32- or 64-bit. This makes Small bytecode non-portable. I asked the author to fully repair this and he declined. Small was targeted primarily toward embedded systems (it has been used in the iPod firmware, for example), and those are always specific to a certain processor, so it's not an important goal to him I guess.

Yeah, that's just stupid. The bytecode generated should be system-independent - that's what the VM is for, to provide a substitute for the system (in essence).

Graf Zahl said:

You know ZDoom's catchphrase, 'WFDS' right?

That said, I already started working on a compiler/interpreter *twice* and got shot down by the big master each time - so I'm sorry I won't put any more work into it - because I know where it will end.

GZdoom split!

Or... If Randy sees that every other active Doom port uses Grafcode (or whatever you're going to call it) he might actually think "Hmm you know what, ZDoom needs this".

Cutman said:

GZdoom split!

Or... If Randy sees that every other active Doom port uses Grafcode (or whatever you're going to call it) he might actually think "Hmm you know what, ZDoom needs this".

And just because it was Graf's work, which we know nothing about, automatically means it was suitable for use? I'm sure Randy had reasons for his response.

Cutman said:

Grafcode (or whatever you're going to call it)

Grafflescript? :o)

Vermil said:

And just because it was Graf's work, which we know nothing about, automatically means it was suitable for use? I'm sure Randy had reasons for his response.

It was openly posted on the ZDoom forum. And BTW, Randy's sole reason was 'I'd like to do it myself'.

How many ports would actually use this new scripting system? Eternity, GZDoom (if it wants to split from ZDoom), EDGE (are major features still being added to this?), and possibly Doomsday?

I realize I am very much out of my element here, but from what I've seen of the Doom3 script language is that it might be something of what would fit in with that criteria. It's compiled at run time, it's really the thing doing everything in Doom 3, (coupled with the DEF files, which are similar in spirit to Decorate/EDF/DDF). It's C based.
I don't really get the 4th criteria so I can't say anything about that though.

Problem is that it'd have to be reverse engineered since the source isn't released just yet.

Graf Zahl said:

And BTW, Randy's sole reason was 'I'd like to do it myself'.

I'd respect that if it were true. But it seems quite blatant that it's false. Randy doesn't want to do it himself, or he'd be working on it. Instead, he has been fumbling around with drawing functions and sound & music APIs.

Honestly, I think you should say that you'd like to do it yourself and split GZDoom from ZDoom in order to develop "GraffleScript".

Because it's simply never going to happen otherwise.

david_a said:

How many ports would actually use this new scripting system? Eternity, GZDoom (if it wants to split from ZDoom), EDGE (are major features still being added to this?), and possibly Doomsday?

I very much doubt that it is even possible for Doomsday and ZDoom to use the same scripting language given the fact that the two are designed so completely at odds with each other.

skyjake has been working on and off on a proprietary, engine-native scripting system for a couple of years which was originally destined to appear in Hawthorn (Doomsday 2.0) one day. Last I heard he had the vm and a virtual threading system up and running but that was six months ago.

Gez said:

Because it's simply never going to happen otherwise.

I suspect the same. However, for the next 3 months things don't look good (work related stress if you understand what I mean... :( )

david_a said:

How many ports would actually use this new scripting system? Eternity, GZDoom (if it wants to split from ZDoom), EDGE (are major features still being added to this?), and possibly Doomsday?

The only port that is close enough to ZDoom in terms of internal design is Vavoom. All others, I fear, would simply not be capable to handle a scripting language that's tied into ZDoom's class hierarchy.

Why not embedded-something?

There is a lot of good scripting languages. Python can be embedded, for example.

It has nice data structures and forces you to write legible (indented) code.

Just an idea... y'know.

shinobi.cl said:

Why not embedded-something?

There is a lot of good scripting languages. Python can be embedded, for example.

It has nice data structures and forces you to write legible (indented) code.

Just an idea... y'know.

Graf Zahl said:

Well, most scripting languages I investigated didn't even come close to the feature set I'd like to have for ZDoom or I would have used one already. The area where most languages are insufficient is with class hierarchy features - something that's essential to expose ZDoom's internal features. Without any simple way to get to the internal classes and data structures (and being able to inherit from them) the languages have already failed to meed the mminimum requirement I set so I fear that a specific implementation is inevitable.

Well, Python has class inheritance. I wonder what specific feature Graf is missing.

A little reference of Python inheritance
http://sysop.com.cn/document/develop/byteofpython_html_120/byteofpython_html/inheritance.html

I'm a programmer, but not a brilliant one... so i guess i just keep reading this thread until i can make a relevant post.... But i still think python may be good.

I'm certain Graf has already investigated Python.

Graf Zahl said:

The only port that is close enough to ZDoom in terms of internal design is Vavoom. All others, I fear, would simply not be capable to handle a scripting language that's tied into ZDoom's class hierarchy.

While getting a language that *everybody* supports would be impossible don't forget that ZDoom basically counts as 3 different ports right there. (Zdoom, GZDoom, and Skulltag) Odamex and ZDaemon while based off Zdoom don't normally keep up with the latest version so it's hard to know if they would be able to support it, but it's theoretically possible. Having 3-4 different ports support the same scripting language is about as close to a 'standard' as you could hope to get.

All this being said, wad authors have still only really begun to tap the power that the current incarnations of ACS and DECORATE provide. When you use these 2 features together it's pretty impressive what you can create with enough imagination. They most definitely have their limits but let's not knock these incredible scripting languages and the effort that went into creating them.

Any port could use a language that was designed properly, to maintain some level of separation between the actual core of the language, and the APIs supported inside that language. That's my opinion, anyway.

This suggestion was to deal only with the mechanics of a language. Parsing and code generation. Not with APIs, certainly not to suggest that ports using the same language would have scripting compatibility with each other. That would be akin to trying to create the One Port that supports all other ports' features.

However, I myself noted that it's unlikely to happen, especially since Randy is content to play in his own yard, so to speak. EE is currently reexamining some embeddable languages, but there are going to be significant difficulties with any of them. For example, Python doesn't support our compiler of choice any more, so if we end up going with it, we're probably going to have to upgrade.

Aabra said:

While getting a language that *everybody* supports would be impossible don't forget that ZDoom basically counts as 3 different ports right there. (Zdoom, GZDoom, and Skulltag)

On the other hand, the most likely scenario will be that Graf will implement GrafScript in GZDoom, and the GrafScript code with be GPL. Randy will not backport it into ZDoom, and Skulltag will not be able to import it.