New source port: d2k

That's a good point. I could imagine how various equivalents would work purely in Lua, but I guess it would be a strange way to go about it.

I'll guess we'll see if/when we get a mature port with a "real" embedded scripting language. I never checked out how mature the Lua in Sonic Robo Blast was. But from my brief foray into this stuff, I definitely feel that a decent embedded language (Lua, JS or whatever) could be used to achieve the same things that other ports use DECORATE, SBARINFO, MENUDEF, etc. for. The definition-versus-programming language distinction isn't really relevant, here. After all, huge chunks of Doom data (state tables, etc.) are defined in C.

Jon said:

I definitely feel that a decent embedded language (Lua, JS or whatever) could be used to achieve the same things that other ports use DECORATE, SBARINFO, MENUDEF, etc. for. The definition-versus-programming language distinction isn't really relevant, here. After all, huge chunks of Doom data (state tables, etc.) are defined in C.

Sorry, but that's plain wrong. A definition language's purpose is to abstract the underlying data and make it more accessible to the modder. A programming language is the tool to implement this abstraction. These are two entirely different layers, if this gets mixed up I can outright promise that the attempt will fail, because the need to know technical internals is not something you want to expose to a mapper.

Of course you can define a new actor directly in C++ (or Lua) instead of DECORATE but you will lose all the ease-of-use and convenience of the underlying parser doing the dirty work for you. If you want to do it in Lua, the result may look like Doom's original info.c. DECORATE's purpose is to HIDE these internals to the maximum degree possible. And that's a lot of dirty work you can't see.

The same is true for the other formats as well. There is no 1:1 correllation between what is in the definition file compared to what ends up in memory internally.

Graf Zahl said:

Sorry, but that's plain wrong. A definition language's purpose is to abstract the underlying data and make it more accessible to the modder. A programming language is the tool to implement this abstraction. These are two entirely different layers, if this gets mixed up I can outright promise that the attempt will fail, because the need to know technical internals is not something you want to expose to a mapper.

It is possible for a single language to do both.

Lua specifically was actually born from declarative configuration langauges. You can read some of the history here and the release notes for the first version of Lua here which specifically extols the virtues of Lua as a configuration langauge. To that end, it contains syntax sugar that allows it to continue to be used in such a fashion. Premake 4 is a good example of this:

-- A solution contains projects, and defines the available configurations
solution "MyApplication"
   configurations { "Debug", "Release" }

   -- A project defines one build target
   project "MyApplication"
      kind "ConsoleApp"
      language "C++"
      files { "**.h", "**.cpp" }

      configuration "Debug"
         defines { "DEBUG" }
         flags { "Symbols" }

      configuration "Release"
         defines { "NDEBUG" }
         flags { "Optimize" }

Behind the syntax sugar, those are all function calls.

There is loads more that you can do to make Lua a good DSL which you can read about here

Yep. In fact, we plan to move the thing definitions into Lua. Dunno about DECORATE/EDF, or ACS. It's probably easier to do it all in Lua, but we'll see.

We evaluated a bunch of languages: Ruby, Python, Lua, JavaScript, and Squirrel, and we stuck with Lua because:

- It's fast
- It's small
- It's in C
- It has bindings for popular libraries (Pango, Cairo, databases, etc.)
- It's simple to read, write and generally understand
- It can be easily and definitively sandboxed
- You can use it for declarative configuration
- Creating bindings is hella easy; the main scripting engine file is < 800 lines

Major downsides were:

- Slightly weird syntax (~= is not equals, car:honk() translating into car.honk(car), etc.)
- 1-based indexing
- Some lack of basic functionality (no built-in way to print a table, etc.)
- Debugging can be less than fun (weak error messages, no types...)

However, we're moving the config from the old format to JSON instead of Lua. The main reason is that writing Lua back into a config file once it's loaded is a big pain. I'm not wild about this though; maybe I'll do a little more reading about serializing it.

Honestly, we narrowed things down to Lua, JavaScript and Python pretty quickly. Python got ruled out because of the whitespace thing; that's difficult to work with in consoles. I actually wanted to switch to JavaScript; I had implemented a few things both in Lua and JS and the JS version seemed a little more structured... probably because it looks a lot more like C. The main thing though was that running D2K on the web becomes a million times easier when big chunks of it are already in JS. Too bad the only JS engines that exist are C++ now, and they force you to compile your project as C++. That's a dealbreaker for us; I'm actually really disappointed there's no leading-edge JS engine in C. It's funny what we end up caring about, haha.

Oh and that reminds me. I think it's fair to wonder why we're bothering with D2K when its purpose is essentially overshadowed by (G)ZDoom, EE, PrBoom(+) and Odamex, and why we chose to base the project on PrBoom+. Basically for us it came down to:

- Written in C
- Has hardware (OpenGL) renderer
- Crazed compatibility, vanilla, Boom, etc.

We hope that PrBoom+'s strengths, plus c/s netcode, Lua scripting, and a few advancements, add up to a port that's fun, accessible, and easy to extend. But hey, gotta fix multiplayer sound first. Maybe tomorrow :)

You mentioned that you wanted to drop the netcode into Eternity once it's stable enough. How long do you think that will be at this point? How will it affect Eternity demo recording?

Ladna said:

We evaluated a bunch of languages: Ruby, Python, Lua, JavaScript, and Squirrel, and we stuck with Lua because:

I'm (very slowly) looking at this too. I haven't ruled out JS yet, which I think is stronger on the nice-language-to-write-in, and possibly also transferrable-skills-from-other-stuff. Quasar said that when he and/or Kaiser looked at it last, GC performance was a big issue. I don't know which JS engine they were looking at. How's Lua's GC?

Ladna said:

- It can be easily and definitively sandboxed

That's very important :>

Ladna said:

However, we're moving the config from the old format to JSON instead of Lua.

Have you considered YAML?

Ladna said:

Too bad the only JS engines that exist are C++ now, and they force you to compile your project as C++. That's a dealbreaker for us; I'm actually really disappointed there's no leading-edge JS engine in C. It's funny what we end up caring about, haha.

I hadn't realised that; that's a big issue for me too. I really CBA with C++.

Ladna said:

- Written in C

Sorry to ask, but what's the deal?
If you do not like OOP you can still convert all your code to C++ with a few changes where stricter type checks will be needed.

But you'd have the advantage to use other C++ code if necessary. With C you'd be stuck, unless the code was contained enough to be encapsulated in a wrapper module.

Personally, I haven't bothered with C for the last 15 or so years, the language has no advantage at all over C++, even in the most limited scenarios C++ will win due to stricter type checks.

Graf Zahl said:

Sorry to ask, but what's the deal?

This sounds a lot like coding fundamentalism, if you ask me.
Like the entire agenda of this port. I think it can be summed up as "Let's gobble up all the greatest features and then implement them in the most conservative way imaginable.

Whether this will work out remains to be seen. I'll be honest here. What I read so far sounds ambitious on one side, but the self-imposed restrictions that are evident here don't raise my hopes that the project will catch on. At the very best it may end up PrBoom++ (without the real '++', of course... :D) and fill precisely the same niche.

But Lua? Lua may be easy to use but what bothers me about it is its pervasive 'made for dummies' attitude with turning sane programming conventions on their head and working differently, just for the sake of being different in a way that may appeal to the uninitiated.

Danfun64 said:

You mentioned that you wanted to drop the netcode into Eternity once it's stable enough. How long do you think that will be at this point? How will it affect Eternity demo recording?

I don't want to put a date on it. Vaporware and over-promising is enough of a problem as it is.

Hopefully c/s code won't affect EE demo recording at all. I imagine the c/s demos for EE will be different than those in D2K; we're planning on using compression and some other stuff and I don't know Quasar's views on additional libraries or how he'd like the c/s demos structured. But again, it's a ways off.

Jon said:

I'm (very slowly) looking at this too. I haven't ruled out JS yet, which I think is stronger on the nice-language-to-write-in, and possibly also transferrable-skills-from-other-stuff. Quasar said that when he and/or Kaiser looked at it last, GC performance was a big issue. I don't know which JS engine they were looking at. How's Lua's GC?

I don't have super hard numbers. We run Lua's GC whenever we render a new frame and it hasn't resulted in a significant FPS drop (if any). And it's not like we're doing nothing; we build a lot of PangoLayout instances for the HUD, for example. When I get some time later today I'll run a quick benchmark.

Jon said:

Have you considered YAML?

There's a lot to like about YAML, in particular it tends to "just make sense" when you look at it, which is a big plus for any configuration file format. We were "eh......." about the complexity and indentation sensitivity though. FWIW, we're not super happy with JSON either. We're going to send data back and forth from the master in JSON, so we'll need that functionality anyway, I just thought if nothing else I'd cut down on the dependencies. I read some about Lua serialization and we might go that route; it turns out you can even serialize functions back to text, so, that's pretty great actually. We want client configs to be scriptable (like, let clients provide their own function to determine preferred weapon order, for example), so that's a big plus.

Graf Zahl said:

Sorry to ask, but what's the deal?

Oh, super happy to discuss, no worries :)

Graf Zahl said:

If you do not like OOP you can still convert all your code to C++ with a few changes where stricter type checks will be needed.

But you'd have the advantage to use other C++ code if necessary. With C you'd be stuck, unless the code was contained enough to be encapsulated in a wrapper module.

Personally, I haven't bothered with C for the last 15 or so years, the language has no advantage at all over C++, even in the most limited scenarios C++ will win due to stricter type checks.

The best thing about C++ over C by far is the stricter type checking. I actually spent a little time exploring converting D2K to C++, but the changes involved were kind of daunting, specifically the action pointers for thinkers.

The rest is, as Hell Theatre points out, essentially just religion. I think OOP is (mostly) silly (even though D2K's HUD widgets use it); I think a better type system isn't worth the potential for over engineering, feature abuse, and general complexity C++ allows; I'm super irked whenever I see "#ifdef __cplusplus" gumming up C code; operator overloading is confusing; C compilers and linkers are significantly faster; blah blah blah. Again, I understand I'm a little irrational about this :)

Hell Theatre said:

Like the entire agenda of this port. I think it can be summed up as "Let's gobble up all the greatest features and then implement them in the most conservative way imaginable.

We disagree that there's a fundamental tension between demo compatibility and advanced features, and we think that ports like EE, PrBoom(+), and Odamex have proven that it's definitely possible to include advanced features without losing compatibility. Is it easier to break compatibility and do whatever you want? Of course it is. But as AlexMax pointed out, demo compatibility ensures that we provably retain the physics and gameplay feel of Doom, which is the main reason we love the game.

Hell Theatre said:

But Lua? Lua may be easy to use but what bothers me about it is its pervasive 'made for dummies' attitude with turning sane programming conventions on their head and working differently, just for the sake of being different in a way that may appeal to the uninitiated.

I'm with you on this. The biggest offender is 1-based indexing, but their weird for-loops are a close second.

But our goal with the scripting interface is to provide mappers and modders access to all parts of the engine without having to be C programmers. What if you want to add double-jump capability? What if you want to add Q2's grappling hook? What if you have an idea for a new game type? These days (unless there are things in ZDoom I don't know about, which is totally possible), you're looking at picking a source port and modifying the engine in C/C++, and these days that's Zandronum. Adding functionality to Zandronum is HARD, first because modifying a Doom engine is hard, and second because there are so many netcode issues to consider.

So yeah, as a professional C developer, Lua's idiosyncrasies irk me sometimes. But I see its appeal to non-programmers and beginner/amateur programmers as a plus. And besides, the fact is Lua is industrial strength; it's used in tons of games with millions of users. 1-based indexing just isn't a big enough sin to override all the pros.

Have you ever looked at http://www.angelcode.com/angelscript/ ? IIRC Kaiser decided it was superior to Lua for his purposes in his Kex 2 engine.

Ladna said:

The rest is, as Hell Theatre points out, essentially just religion. I think OOP is (mostly) silly (even though D2K's HUD widgets use it); I think a better type system isn't worth the potential for over engineering, feature abuse, and general complexity C++ allows; I'm super irked whenever I see "#ifdef __cplusplus" gumming up C code; operator overloading is confusing; C compilers and linkers are significantly faster; blah blah blah. Again, I understand I'm a little irrational about this :)

Hm. I can't say I agree on this. I absolutely hate the overengineering that seems to popular among OOP programmers just as much, and yet I haven't looked back ever since starting to work with C++.
Especially Doom's entire thinker system is a textbook case for perfect OOP suitability, and I think ZDoom proves that part rather well, even though even ZDoom has its overengineered parts, the worst culprit being the type system that just got implemented for the scripting. I am dead certain it could have been done with half as much code and overhead, but that piece of code was out of my hands...

The tool itself is not the problem, but the way it has been misused certainly is. 90% of all C++ code I have seen suffers from overengineering, but on the other hand most modern C code I see suffers from convolution that mostly is the direct result of avoiding OOP. I prefer the middle ground where I can use the good parts from both worlds. In fact, I think Eternity got the balance between OOP and procedural code quite right - it might have been able to do a bit more if it wasn't for demo compatibility.

We disagree that there's a fundamental tension between demo compatibility and advanced features, and we think that ports like EE, PrBoom(+), and Odamex have proven that it's definitely possible to include advanced features without losing compatibility.

It may all be possible, for sure, but I wonder if, for example, ZDoom's complex inventory system can be done in a more 'compatible' manner, with powerups not being some index in a table but an actual actor in an inventory chain, using its C++ class methods to do its magic.

But our goal with the scripting interface is to provide mappers and modders access to all parts of the engine without having to be C programmers. What if you want to add double-jump capability? What if you want to add Q2's grappling hook? What if you have an idea for a new game type? These days (unless there are things in ZDoom I don't know about, which is totally possible), you're looking at picking a source port and modifying the engine in C/C++, and these days that's Zandronum. Adding functionality to Zandronum is HARD, first because modifying a Doom engine is hard, and second because there are so many netcode issues to consider.

I'd say choosing Zandronum as the base for modification is the first step to guaranteed failure. I know how messed up the netcode is it inherited from Skulltag. The engine is littered throughout with network helper calls because it can't just operate on object state alone, and for anyone trying to implement something new this will inevitably develop into a roadblock that's nearly impossible to overcome.

So yeah, as a professional C developer, Lua's idiosyncrasies irk me sometimes. But I see its appeal to non-programmers and beginner/amateur programmers as a plus. And besides, the fact is Lua is industrial strength; it's used in tons of games with millions of users. 1-based indexing just isn't a big enough sin to override all the pros.

Good to see that we can agree here. But one thing I utterly fail to understand why languages that put so much semantics into English-language keywords are considered beginner-friendly.

From personal experience, the concept of braces is far easier to convey than to have people learn that a block starting with 'if' has to end with 'endif', or 'for' with 'next' (or whatever else a language defines for its structures.) In C-like languages, when trying to explain such stuff is "Here's a brace that starts a block, how do you close it?", the answer is nearly always "with a closing brace, of course". The semantics are implicit and I've yet to find a person who didn't grasp that concept quickly.

(Well, thank God it's not Cobol... :D)

Linguica said:

Have you ever looked at http://www.angelcode.com/angelscript/ ? IIRC Kaiser decided it was superior to Lua for his purposes in his Kex 2 engine.

That one escaped our notice when we were deciding on engines. I looked it up a little while catching up on EE's Aeon stuff, and it's pretty nice. It looks like a regular programming language, it's fast, strongly typed, and used basically everywhere (that usage list is long, haha). It's got a lot of cool, "pro" features, like deterministic execution and debugging stuff too. I wouldn't be disappointed if I had to use it. I'm not sure how easy it is to hook into other libraries though. We rely on bindings to Pango, Cairo and friends, we'll probably end up using more, and it's nice to not have to write those. It looks like a good choice though, as expected :)

If anyone's curious, here's a sample of local_hud_widgets.lua:

d2k.widgets = {}

local messages_widget = RetractableTextWidget.RetractableTextWidget({
    name = 'messages',
    z_index = 1,
    top_margin = 4,
    bottom_margin = 0,
    left_margin = 8,
    right_margin = 8,
    width = 1,
    bg_color = {0.0, 0.0, 0.0, 0.0},
    vertical_alignment = TextWidget.ALIGN_BOTTOM,
    outline_color = {0.0, 0.0, 0.0, 1.0},
    outline_text = true,
    outline_width = 1,
    line_height = 4,
    use_markup = true,
    font_description_text = Fonts.DEFAULT_MESSAGES_FONT,
    strip_ending_newline = true,
    retractable = RetractableTextWidget.RETRACT_UP,
    retraction_time = 500,
    retraction_timeout = 2000,
})

messages_widget:set_external_text_source(
    d2k.Game.get_consoleplayer_messages,
    d2k.Game.get_consoleplayer_messages_updated,
    d2k.Game.clear_consoleplayer_messages_updated
)

messages_widget:set_parent(d2k.hud)
d2k.widgets['messages'] = messages_widget

This is how we instantiate the widget that displays game messages. Admittedly there's not a lot going on, but that's hopefully the point: with a small amount of code, you can have a lot of control over your HUD. You can also exercise that control through the console, so like:

> d2k.widgets['messages'].set_font_description_text('Comic Sans 10')

And so on.

You can probably also see why documentation will be a big part of the scripting engine effort. Constants like TextWidget.ALIGN_BOTTOM, accepted constructor arguments, etc. all need to be documented. We'll get there :)

Graf Zahl said:

Hm. I can't say I agree on this. I absolutely hate the overengineering that seems to popular among OOP programmers just as much, and yet I haven't looked back ever since starting to work with C++.

That's fair, and this kind of thing is possible in pretty much any language.

Graf Zahl said:

Especially Doom's entire thinker system is a textbook case for perfect OOP suitability, and I think ZDoom proves that part rather well

I more or less agree, largely because C Doom engines use type punning for mobj_t/thinker_t "inheritance".

Graf Zahl said:

The tool itself is not the problem, but the way it has been misused certainly is. 90% of all C++ code I have seen suffers from overengineering, but on the other hand most modern C code I see suffers from convolution that mostly is the direct result of avoiding OOP. I prefer the middle ground where I can use the good parts from both worlds. In fact, I think Eternity got the balance between OOP and procedural code quite right - it might have been able to do a bit more if it wasn't for demo compatibility.

Agreed here too, though I think EE's demo compat handling is maybe the cleanest I've seen.

Graf Zahl said:

I'd say choosing Zandronum as the base for modification is the first step to guaranteed failure. I know how messed up the netcode is it inherited from Skulltag. The engine is littered throughout with network helper calls because it can't just operate on object state alone, and for anyone trying to implement something new this will inevitably develop into a roadblock that's nearly impossible to overcome.

I'm kind of in awe at Zandronum's success. If you described the project to me, I'd give it a 5% chance of working at all. It's a testament to its devs and community that it not only works, that it's the de facto c/s Doom experience.

Graf Zahl said:

Good to see that we can agree here. But one thing I utterly fail to understand why languages that put so much semantics into English-language keywords are considered beginner-friendly.

It's funny what you learn about something after spending a lot of time with it. Lua is really light on punctuation; as you noted, blocks are terminated with words, there are no pointers or sigils, conditionals and loops don't use parentheses, etc. At first you think, "well, how accessible and simple, if a little verbose", but as time goes on, you realize that because EVERYTHING is a word, your eyes can't quickly pick out blocks as well as they could have using braces. It has the vague effect of turning everything into mush in a way. It was something that pushed me towards JS, definitely. But I think this is (humbly speaking) a programmer power feature, or maybe just habit. Like, I've written/read/modified hundreds of thousands of lines of code, maybe millions, I don't know. I can mostly scan a well-formatted code file and pick out questionable code based on whether it "looks wrong" within a few seconds. I'm far slower reading Lua, maybe due to the words-as-block-delimiters thing, maybe due to the fact it's a little more free-form, or it has a lot less punctuation, or I'm just not as used to it. But it's something I've noticed, definitely.

It's one of the things that makes me prefer C code over C++ code. I know this is debatable but, C code is far more uniform than C++ code is, which makes scanning it and understanding it a much more automatic process. Style guides and limiting C++'s features can do a lot about this (don't use the STL, don't use exceptions, keep class method definitions in a single file, etc.) and C can definitely get out of control (WTF is this global state variable doing hidden here at line 1157?!), but this is just my experience.

That said, I don't think the intended audience will care or even notice. Probably looking like C is a turn off, judging from the popularity of languages like Ruby and Python anyway.

Graf Zahl said:

(Well, thank God it's not Cobol... :D)

Hmmmm, should we evaluate embedding cobol....? :P

Ladna said:

It's funny what you learn about something after spending a lot of time with it. Lua is really light on punctuation; as you noted, blocks are terminated with words, there are no pointers or sigils, conditionals and loops don't use parentheses, etc. At first you think, "well, how accessible and simple, if a little verbose", but as time goes on, you realize that because EVERYTHING is a word, your eyes can't quickly pick out blocks as well as they could have using braces. It has the vague effect of turning everything into mush in a way.

Precisely my problem with such languages.

But still, what I find even more confusing is that this is considered beginner-friendly. The concept of such a language is a lot harder to teach than a simple idea like 'a block begins with a brace ( '{' } and ends with a brace ( '}' )'. Any time I had to explain such concepts the C-version was the easier one that far more people understood quickly. The one single concept of C that's hard to understand is 'for' loops, because they are done in a rather technical way that may escape some people.

I'm reminded of this.

Lol, yes. What a hero that guy was.

Hell Theatre said:

This sounds a lot like coding fundamentalism, if you ask me.

These are hobby projects. Why should anyone be forced to use something they don't want to? For my own stuff, I understand C, I can read and write C, I can make a quick assessment as to whether any given C library I might consider using is well written and suitable or not. I really dislike reading or writing C++. So why should I subject myself to it?