Considering Lua

[Quasar]

I have been considering Lua for integration into Eternity but I have found several problems with the language that I feel limit its usefulness:

1. No named constants. There is no form of enumeration, no form of #define, or anything else to provide pre-defined values for the user. This results in being required to use magic numbers throughout the code.

2. No bitwise operators. This means no ability to combine or test for bit flags, resulting in a proliferation of separate, inefficient boolean values instead.

3. No intrinsic support for unsigned integer or fixed point math. Fixed point could be defined by adding a library, but the effort required to program it and the way it works for the end user are unsatisfactory to say the least.

Upon examination, I question why Lua has become the defacto standard for game scripting. While it may be fast, compact, and stable, which are all important, the language implementation remains toy-like due to missing all of these important features. I certainly would not be moving any native code into it, as I had hoped would be possible. It would end up as a horrible mess.

[andrewj]

Quasar said:

1. No named constants. There is no form of enumeration, no form of #define, or anything else to provide pre-defined values for the user. This results in being required to use magic numbers throughout the code.

Oh rubbish, this taken straight from Oblige:

-- thing flags
MTF_EASY    = 1
MTF_MEDIUM  = 2
MTF_HARD    = 4
MTF_AMBUSH  = 8

2. No bitwise operators. This means no ability to combine or test for bit flags, resulting in a proliferation of separate, inefficient boolean values instead.

You could add it yourself, just implement some native functions like bitand(), bitor(), bitneg().

Or you could do it the "lua way" which is using a table and the presence/absence of a keyword is your bitflag. Example:

flags = {}
flags.foo = true  -- set the bit
flags.foo = nil   -- clear the bit
if flags.foo then -- test the bit

3. No intrinsic support for unsigned integer or fixed point math.

You don't need fixed point math in the language. You would convert from fixed point to floating point (and vice versa) in all of your native functions.


The real issue for me of using Lua for game scripting is persisting and restoring the lua vm state for savegames. Persistence is not part of the core lua library, there is a 3rd party library (called Pluto or Jupiter or something) which doesn't seem finished, and I'm certain that when a problem occurs, debugging that stuff would be really really hard (requiring a lot of knowledge about Lua's internals).

[Quasar]

Problem 1: Those aren't constants. They're just normal global variables. This is a minor issue, of course. The user would know better than to overwrite the values in those variables. However, you could certainly overwrite them by accident :P

Problem 2: Calling native functions just to do bitwise ops is a bit inefficient and kind of ugly, especially if you need to combine N values at once, rather than having to add one flag at a time.

Problem 3: The code must remain in fixed-point if I am to convert native code (mainly, the action functions) into Lua. Why? Compatibility. Floating point math does not render the same mathematical results as 16.16 fixed point.

However, fraggle has suggested that using hashed or otherwise looked-up string values for constants and flags is a good possibility and he doesn't believe it will have a true performance impact. I've agreed to try implementing them this way if I go ahead with Lua support.

Thanks for your input and ideas.

EDIT: There seem to be some fairly trivial solutions to saving and loading, actually. Because like Small, Lua cannot suspend execution in the middle of the script, there is no need to save things such as the stack, registers, pc, etc. Only the global variables need to be archived, and it just so happens that Lua is based on associative arrays and therefore keeps all of the globals in one of these. It should suffice to iterate over it and save out all variables inside/restore them to their values by name on load.

[fraggle]

Quasar said:

2. No bitwise operators. This means no ability to combine or test for bit flags, resulting in a proliferation of separate, inefficient boolean values instead.

Good.

You should not be making people scripting levels learn bitwise logic in order to do something simple like turn a thing's flag on and off. Bitwise logic is something that even a lot of programmers don't seem to understand properly, and the idea that you should make level editors do the same thing is absurd. When I say "a lot of programmers", I mean that I know experienced programmers that I've worked with in a professional capacity who don't know this stuff. Disgraceful? Yes, but it gives you a feel for how non-intuitive this stuff is.

I specifically dealt with this when I wrote FraggleScript with the "objflag" function to get/set the value of a particular flag.

Furthermore, doesn't Eternity have a secondary "flags2" field? How would you deal with that? Force everyone to learn which flag goes in which field, presumably?

[andrewj]

Quasar said:

Problem 3: The code must remain in fixed-point if I am to convert native code (mainly, the action functions) into Lua. Why? Compatibility. Floating point math does not render the same mathematical results as 16.16 fixed point.

Yes I can see that is a problem for a port that retains a high degree of DEMO compatibility.

While forcing Lua to use 16.16 fixed point math is surely possible, it would be a lot of work, hence you'd probably want to keep the old functions in C/C++ and only use Lua for new stuff.

[esselfortium]

fraggle said:

Furthermore, doesn't Eternity have a secondary "flags2" field? How would you deal with that? Force everyone to learn which flag goes in which field, presumably?

Eternity now has a cflags field that accepts flags, flags2, and flags3 interchangeably, though the codepointers to set or unset flags in a particular frame still require you to know which flags field they came from.

I definitely agree that users shouldn't need to know or care which flags are in which field, ever, though. It's good that cflags makes it no longer necessary in most cases, but it's still a problem for setting/unsetting flags ingame, as it guarantees you have to make another trip to the documentation. :P

[SoM]

See I would prefer something with a syntax closer to C, but that's probably because I code in C... :\

[printz]

I'm with Quasar. I wouldn't accept a language for Doom without constants, bitwise ops, enough data types either. If the current scripting language is to be replaced with something modern, then make sure it looks professional. Also, this is Doom, not just any other NEW game. Doomers who take Doom seriously should be aware enough of computers.

Also, fixed point is where Doom's at, not floating point, as this is not Quake. I guess in fact Doom has absolutely no call to any float or double function. If it has, they're insignificant. No need to prove me otherwise, thanks.

I'd rather search for another language, one that would fit Doom's style enough -- but look clear and not let the coder "compress" it with nested stuff, as in C(++).

That "small" is already typeless, works for me just fine, btw. I don't know how nice would it work for a run-time interpreter (if the script's not compiled), however. So typeless may still be a bad idea.

[TheDarkArchon]

printz said:

Also, fixed point is where Doom's at, not floating point, as this is not Quake. I guess in fact Doom has absolutely no call to any float or double function. If it has, they're insignificant. No need to prove me otherwise, thanks.

This is for good reason: More than a minority of computers could not do floating point maths at the time Doom was released.

[Quasar]

The issue on flags isn't just one of end-user convenience but also of efficiency. Scripted code is anywhere from 10 to 1000x slower than native code, varying dramatically with the structure of the code and the efficiency of the virtual machine and/or ability of the bytecode compiler to do optimizations.

I'm certainly all for a different interface for end users to use flags with. Small in fact has such a function which accepts a STRING of flag names to turn on or off, the same type of strings which are used to specify flags in BEX and EDF.

But if I'm going to convert *native code* into a language, and it does this:

mo->flags &= ~(MF_SOLID|MF_SHOOTABLE);

I'm not going to convert it into a script that has to do this:

RemoveFlag(tid, "SOLID");
RemoveFlag(tid, "SHOOTABLE");

Two tid lookups, two flag string parser calls, and two native function dispatches (containing considerable work for the vm), versus what in a CPU could possibly compile to one single instruction if the register allocation is right for it. This is not the way to go about conversion of native code into a scripting language, unless you want to see the game become dramatically slower. And this is the kind of difference that, multiplied by 1000 monsters on a large map, cannot be made up just by fortitude of your CPU. Even a modern machine would start to feel the lag from too much of that kind of inefficiency.

[printz]

What's your plan with new scripting languages (like Lua)? Do you consider to have them replace Eternity's "small" language in the same way FraggleScript was removed? Is it planned to be an interpretable language? I guess machine-code is run faster than read text, however. And "small" is fun enough as it is, because it's typeless. Only drawback I'm seeing is that, as you mentioned, it's not decompilable.

[Quasar]

Small will stay in, but will be considered deprecated due to the problems it has. This means you can still use it, but I won't be encouraging its use since it may result in wad files with a limited lifetime or portability.

[david_a]

I don't see why your scripting language can't abstract away some of the complexity of bitwise logic without turning it into something dramatically slower. Something like this (psuedo-C#-ish code):

myMonster.Flags.TurnOn( ActorProperty.Solid, ActorProperty.Shootable );
myMonster.Flags.TurnOff( ActorProperty.SomeOtherFlag );

Of course if you're writing the language/compiler yourself it's probably no fun to support all the things needed for the above snippet...

[andrewj]

I just noticed that the Pluto serialization library has become active again (in January) when the previous version was in 2005. Looking at it once more, and having thought through some of the issues, serialization no longer worries me to use Lua as a full scripting language in a DOOM game engine. [Speed still worries me].

The point I wanted to make in this post though is that something feels rather wrong about adding Lua to a mature full-featured DOOM port like EDGE, Eternity, ZDoom etc... For one thing, they already have some scripting capabilities (though EDGE's RTS is very primitive and "toy" like). Another thing is that DDF/EDF/DECORATE could also be replaced with Lua tables, possibly used directly by the Lua scripting instead of parsed into C structs, which would make the engine as a whole much simpler / more consistent. Breaking everybody's mods would not be very popular :), but this seems like the Right Thing for a port with Lua at the core. Lastly in the case of Eternity, I think maintaining demo compatibility will be very difficult if large parts of the C code are converted to Lua code.

[SoM]

david_a said:

myMonster.Flags.TurnOn( ActorProperty.Solid, ActorProperty.Shootable );
myMonster.Flags.TurnOff( ActorProperty.SomeOtherFlag );

you know what would be even better?

flagSet(actor, mf_solid, mf_shootable);

if(checkFlag(actor, mf_solid))
{
   //BLAH
}

Often, the fewer letters a script writer has to type the better.

[david_a]

I guess if you're typing it all by hand. I'm used to VS doing most of the grunt work. I suppose having full IDE support for any scripting language is probably a stretch unless an existing established language is used, however.

[Quasar]

Right now it is looking extremely likely that EE will be integrating a highly modified version of QuakeC. QuakeC was designed for the exact purpose of specifying entity state logic handlers, and has been refined and enhanced through several generations of QCC compiler development and Quake source ports.

The particular package I am currently looking at is called FTEQCC. It has integrated virtually all of FrikQCC's enhancements as well as many of its own.

Some of the obstacles to QuakeC integration are converting that language's compiler to function on buffers instead of files (a minor change), ripping out its frame specification stuff (EE does this better and in a Doom-suited way with EDF), and most importantly, replacing the entity reference system with an mobj reference system, which by necessity must function differently. Needless to say, this stuff isn't something that will happen overnight, since I am alone in working on it.

[andrewj]

Interesting... Given that QuakeC was designed for an FPS game (by Carmack no less) it should fit in pretty well.

Are you making it compile on-the-fly (when a wad is loaded), rather than making users compile the scripts themselves?

[Quasar]

That is what I want to do, which is why the compiler will need to be rewritten to work with memory buffers instead of files :)

[printz]

Sounds like Ultimate Power. But doesn't VaVoom already have a language? maybe you can take ideas from them, into this more vanilla-friendly engine no less.