A word about dealing with problems in ZDoom

Arctangent said:

( fun fact: you can actually write the entirety of an actor in a single line with decorate - hell, you can do this with multiple actors )

This may be strictly true, but ZDoom's DECORATE parser has a ton of inconsistencies which exist due to backwards compatibility with a time when ZDoom's tokenizer was less than adequate. For example if your state doesn't have an action function then ZDoom requires a new line to terminate it. So not only is it questionable to do it, but it's actually quite a pain to do it in ZDoom as it stands.

Of course well written DECORATE as-is can be parsed with a one token look ahead parser, and ECWolf does this. The only ambiguous point is an action function call with no parameters. This, however, can be resolved by not allowing 4 character action function names (what ECWolf does), requiring empty parens, semicolons (what ZScript is currently doing), or new lines (what ZDoom kind of does for DECORATE).

So why the semicolons?

Here's the reasons:

1. The lack of proper separator syntax made it nearly impossible to change the properties parser to accept constant expressions that are not a numeric literal. It was such a messy situation that nobody ever went there. So effectively there's 3 choices: Leave everything as it was, including the artificial limitation, add some assignment token or add a separation token. I chose the separation token because it's the least ambiguous here.

2. The states parser may look fine on the surface but internally it had become a huge clusterfuck. This was all because the initial implementation was a lot more limited and had a somewhat different syntax and had no concept on expressions as function parameters. All this stuff was stacked upon a very limited parser over time and it's surprising that it never broke down. But ultimately it very unintuitively depended on line breaks that were handled differently in different contexts, causing all sorts of internal problems.

3. Most importantly: ZScript is not an ad-hoc parser like DECORATE but uses a formal grammar. And the 'formal' aspect is important here. With this approach some things have to be done a little differerently to avoid parsing conflicts.

I am surprised that this comes from an actual programmer who should know his stuff.

Oh - and if someone can tell me how to change the state parser to work without them, I'd happily accept that solution - but I know for sure that this isn't going to be easy with a parser generated from a formal grammar. Such a parser lacks the needed info to recognize the sprite name as such - and the 4 character limitation is not really implementable without some gross hacks in the tokenizer - on top of the one that is needed to parse sprite names and frame characters without quotation marks.

dpJudas said:

Come on. You can't seriously be saying that because one port started work on a certain type of feature that no other port is allowed to be inspired by a good idea. If that's the case, I'd like to say that if anyone does anything multithreaded in a renderer, then they are so totally ripping off QZDoom. :)

Sadly some people think so. Let's not forget that the portal stuff I did was mostly spawned by an external code submission for visual line portals that required proper integration into the physics code - what it came with was very simplistic and partially even sloppy, so not something that really could remain. It was just coincidence that this submission came at roughly the same time as Printz started his portal work on Eternity.

And let's be clear about something: If you do not want to share your code, put it under a license that disallows that (not that I acually used any code from Eternity here in the first place) and if you do not want to share your ideas, I'd say that open source development is not the right concept to do this stuff. Open source is entirely about sharing not only ideas and concepts but also the code itself, provided that no licenses are being violated in the process.

Graf Zahl said:

And let's be clear about something: If you do not want to share your code, put it under a license that disallows that (not that I acually used any code from Eternity here in the first place) and if you do not want to share your ideas, I'd say that open source development is not the right concept to do this stuff. Open source is entirely about sharing not only ideas and concepts but also the code itself, provided that no licenses are being violated in the process.

To be clear: I agree with you fully here, but in a devil's advocacy it's too easy to argue you have the "winning" source port - the most popular one, so naturally you do not suffer from the lack of attention others get. (I disagree with the statement I just made - but I wanted to get it out there so that I could preemptively dispel it before it even became an issue)

However, that argument is entirely irrelevant. What is relevant is people do not understand the numerous ways that these engines actually support each other. Portals are not even an "Eternity" or "Edge" idea - if you really want an originator, try the Build Engine. Shadow Warrior had working portals long before any of these source ports even existed. Long before the source code for either game was even released. The execution may be different, but the concept and use cases are exactly the same: Additional and more flexible use of vertical space, plus neat special effects.

In fact, the concept may even go back further than Shadow Warrior, but that is the first game that I can remember that used them.

But to argue that ideas belong to any one source port and should not be given to others is - to say the least - backwards thinking, and really not very productive. That would be like saying client/server code belongs exclusively to CSDoom, and cannot and should not be added to ZDaemon or Skulltag or Zandronum or Odamex no matter their implementation or how different it is because someone else came up with the idea first.

Eruanna said:

To be clear: I agree with you fully here, but in a devil's advocacy it's too easy to argue you have the "winning" source port - the most popular one, so naturally you do not suffer from the lack of attention others get. (I disagree with the statement I just made - but I wanted to get it out there so that I could preemptively dispel it before it even became an issue)

Even risking going off-topic here, I do not think for one minute that Eternity has remained such a niche port because ZDoom is 'stealing' its features.

I haven't been here for long enough to know the entire history, but I believe this is a result of several bad design decisions over the entire development.

First, Eternity has been a DOS only engine for many of its earlier years, making it an instantly unappealing target for everybody do use. The most ludicrous outcome out of this has been that botched Caverns of Darkness release with its Eternity derived custom engine which even for most users at the time had been unplayable right out of the box. At the same time other ports were focussing on Windows already and slowly branching out to other platforms and gaining traction. Eternity needed several more years to get there.

Second - and I think this is the biggest issue these days, is that the Eternity devs deliberately chose to use entirely different definition formats for everything than what had already been established by other ports. So, in order to target this engine, one has to either do double work or make a project exclusive. That may work if you have some market power and you can push your solution through. But if you are the odd business out, like Eternity clearly is, you have to compromise, make it EASIER to add support, e.g. by adopting ZDoom's MAPINFO instead of rolling out a completely incompatible solution just because.

This will instantly drive some modders away and a port without community can't afford that. Now seriously, what would give a tremendous boost to Eternity was if it suddenly could run some early ZDoom mods like Tei Tenga or Dark 7. These are mods that do not go too deep into ZDoom features but they are enough to drive the point home about incompatibilities in engine design that have been entirely deliberate on the Eternity side with no intent to address - the best example would be ZDoom's colored lighting, which is infinitely more flexible than Eternity's clumsy idea to piggyback on the Boom colormap implementation. This is a classic case of choosing a feature for its ease of implementation, not for its ease of use - which simply does not exist.

Even if some people might cry "sell-out!", I think going this route is the only way how Eternity could ever gain some traction.

I think how it went with GZDoom is the best example of how to make a project succeed: Be compatible with something popular and add something to the mix that hasn't been there before. Eternity could do the same - opening up the possibility of playing light-featured ZDoom maps in a demo-compatible engine. That would give users something of value, the port could come out of its niche and then slowly develop its own identity. But by remaining in the niche, too afraid to come out, is never going to work.

Eruanna said:

To be clear: I agree with you fully here, but in a devil's advocacy it's too easy to argue you have the "winning" source port - the most popular one, so naturally you do not suffer from the lack of attention others get. (I disagree with the statement I just made - but I wanted to get it out there so that I could preemptively dispel it before it even became an issue)

Correct so far - but if a port does not get the attention it deserves maybe it is time to do some analysis why this is so. The results may be surprising and not what the developers would expect. And if you work from an incorrect assumption, even the best work won't yield good results.

It was precisely such a thing that ultimately doomed the project I worked on from 2008-2012. That project was a casual gaming suite but whatever the company tried, nothing worked. They poured endless work into making the games better, they added tons of non-gaming features to the thing, but nothing helped. Customer retention was one month at average. When they did a post mortem on the project it turned out that the one thing I was convinced was the cause for this but nobody else believed really did it it: They didn't put their focus on frequently adding more games to keep people hooked in, they wasted all the work on making games look better and adding useless 'social features' to the UI, i.e. everything that was done didn't interest the customers (who wanted to play new games every now and then) and far too little was done on the one thing that may have helped.

I have to agree with Hell Theatre on one point: From the lack of attention Eternity gets it is very clear that something is done wrong to excite potentially new users. I just don't have the slightest idea what that may be and how it can be fixed. If all things were equal, Eternity should be a lot more popular.

ZDoom is Windows, full of hacks and with a weird API hamstrung by decades of backward compatibility requirements. Yeah it might not run stuff exactly like it did before, but it still runs it; and if there's really a problem, don't worry, you'll just get another compatibility kludge grafted on to accommodate the broken stuff.

Eternity is Macintosh, cleaner and more elegant, but sometimes tossing around all baggage without a care for backward compatibility. The result is that you can run dsv4_rvltinE in ZDoom and it'll work better than trying to run it in EE. The biggest issue with Eternity is scripting: first there was FraggleScript, then it was torn out and Small was added, then Small was torn out and Aeon is in the work. Good thing DavidPH came and gave it an ACS VM that can support pretty much all the enhancements and additions of ZDoom's ACS.

ZDoom grew organically, chaotically, haphazardly, but it grew; while Eternity wanted everything to be neat and consistent and therefore postponed growth until the Perfect Design was done first.

So basically Eternity's biggest failings as far as getting a large "market share" are IMO its lack of stability (the scripting situation) and a bunch of things being vaporwared as they wait on something else to happen (Heretic support waiting for EDF to handle defining weapons and inventory items for example; or UDMF waiting on Hexen map format support first).

Or, to put it another way, ZDoom might be too eager (adding new features that, in hindsight, were bad ideas or should have been implemented differently but it's too late to change it now) while Eternity would be too shy (postponing everything to avoid making the same mistakes ZDoom did).

Gez said:

ZDoom is Windows, full of hacks and with a weird API hamstrung by decades of backward compatibility requirements. Yeah it might not run stuff exactly like it did before, but it still runs it; and if there's really a problem, don't worry, you'll just get another compatibility kludge grafted on to accommodate the broken stuff.

Eternity is Macintosh, cleaner and more elegant, but sometimes tossing around all baggage without a care for backward compatibility. The result is that you can run dsv4_rvltinE in ZDoom and it'll work better than trying to run it in EE. The biggest issue with Eternity is scripting: first there was FraggleScript, then it was torn out and Small was added, then Small was torn out and Aeon is in the work. Good thing DavidPH came and gave it an ACS VM that can support pretty much all the enhancements and additions of ZDoom's ACS.

Demos.

By all mean, record a demo of DSV4 in an old Eternity version that had FraggleScript and try to run it in a modern version that doesn't.

Just calling out the hypocrisy of your argument.

Gez said:

Yeah it might not run stuff exactly like it did before, but it still runs it

dpJudas said:

Come on. You can't seriously be saying that because one port started work on a certain type of feature that no other port is allowed to be inspired by a good idea. If that's the case, I'd like to say that if anyone does anything multithreaded in a renderer, then they are so totally ripping off QZDoom. :)

I felt that, due to the timing of events, that, yeah, it was a bit tasteless. Be inspired, write it, but don't release to DVD, before it has played out in the theater, so to speak.

Graf Zahl said:

So why the semicolons?

Here's the reasons:

1. The lack of proper separator syntax ...
2. The states parser may look fine on the surface but internally it had become a huge clusterfuck....
3. Most importantly: ZScript is not an ad-hoc parser...

All very reasonable.

Graf Zahl said:

I am surprised that this comes from an actual programmer who should know his stuff.

No, you're not - you just said that for effect. You know as well as I do that there's more than one way to skin a cat. Semicolons are what you prefer. The same could be done (with slight changes to some syntax) with line breaks, periods, colons, keywords, etc. The fact that DECORATE did not require them is going to be a point of confusion - that was the only real point I was trying to make.

Graf Zahl said:

Oh - and if someone can tell me how to change the state parser to work without them, I'd happily accept that solution - but I know for sure that this isn't going to be easy with a parser generated from a formal grammar. Such a parser lacks the needed info to recognize the sprite name as such - and the 4 character limitation is not really implementable without some gross hacks in the tokenizer - on top of the one that is needed to parse sprite names and frame characters without quotation marks.

Also a good point. The fact that DECORATE allowed unquoted sprite names, followed by unquoted multiple frame characters, as well as function names made DECORATE very tricky to parse, with known failure cases - I get it.

Graf Zahl said:

Sadly some people think so. Let's not forget that the portal stuff I did was mostly spawned by an external code submission for visual line portals that required proper integration into the physics code - what it came with was very simplistic and partially even sloppy, so not something that really could remain. It was just coincidence that this submission came at roughly the same time as Printz started his portal work on Eternity.

I make no accusations. I just try to put myself in their shoes, and can easily imagine the feeling I would get, and it's not pretty. Maybe it is all just bad timing, and an unfortunate set of cimcumstances. Regardless, I think the righteous thing would have been to sit on it for a while, and let due credit happen. This also highlights the lack of feature-based cross-port collaboration, bringing the issue front and center.

Graf Zahl said:

And let's be clear about something: If you do not want to share your code, put it under a license that disallows that (not that I actually used any code from Eternity here in the first place) and if you do not want to share your ideas, I'd say that open source development is not the right concept to do this stuff. Open source is entirely about sharing not only ideas and concepts but also the code itself, provided that no licenses are being violated in the process.

Come on, mate, don't hide behind that speech. I talking about moral obligation, not legal obligation.

Arctangent said:

hahahahahahahahahhahahahahahahahahahahahahahahahahahahhahah

I wonder what that laugh would sound like...

Arctangent said:

( fun fact: you can actually write the entirety of an actor in a single line with decorate - hell, you can do this with multiple actors )

( funner fact: the fact that you can do this makes other things utter ass )

WTF are you mumbling about?

Hell Theatre said:

Even risking going off-topic here, I do not think for one minute that Eternity has remained such a niche port because ZDoom is 'stealing' its features....
...
Second - and I think this is the biggest issue these days, is that the Eternity devs deliberately chose to use entirely different definition formats for everything than what had already been established by other ports.

This is complete nonsense. The only established definition that could be considered a compatible standard is DeHacked and Boom. DECORATE was just as home-grown as EDF, DDF, and a number of others. You mention that to target Eternity, people have to do "double work"? Do you mean ZDoom's format, and some other format? Why is making it use ZDoom's format necessary, if you are targeting Eternity?

Get this: ZDoom's method is as incompatible and exclusive as anyone else approach that came after Boom and DeHacked. Because, Guess what? Doom never had a monster definition language, except for that which was hard-coded into the executable. ZDoom hacked together their formats just like every other port made their own, since no one was willing to commit to the spirit of collaboration.

My God, in this very thread Graf is talking about replacing DECORATE with ZScript, due to it's inherent shortcomings, and you're calling it the de facto standard? Learn the history of the source before you spit out biased nonsense. Amazing.

@Graf: You have a unique opportunity to end this years-long debate, and bring the people and the ports together. I suggest exposing your plans for ZScript to the community. The entire community. You mention open source. Open up your ideas to the community, and allow all source port developers to mold the language. Drop the "Z" out of the name, and use a name that suggests multi-port compatibility. Let's come together and make a scripting language/actor definition language that all ports can use, so, finally, there can be some chance of cross-port compatibility. This would redeem and remedy years of vertical non-unified development. Make it generic, requiring a compatibility layer so it can be bolted on to ZDoom, Eternity, Your Mam's Doom, whatever. Doesn't that make more sense than the one man ZShow?

Can you step up to the plate, and be willing to be wrong, in some cases? Will your ideas hold water with the rest of the development community, and will you accept modifications/enhancements to your plan?

And, Community, will you participate, with the spirit of compatibility and usability, and devote some energy towards finally moving towards a standard more modern than Boom? And, will you be able to move past silly semantics (such as I did with the semicolons), and stay focused on the goal? I truly wish for this to happen. My port is at a point where I seriously need to consider what my final language/object definition code will look like. I will build it from scratch if necessary, and I am prepared to do so. I'd much rather use something to has a chance of working across multiple ports, however.

Wouldn't it be nice to build a custom monster, and have it appear in Eternity, in ZDoom, and in others?

UDMF turned out relatively clean, good and future-proof, right?

Can't something similar be done for the script language? Before it is too late to change course.

kb1 said:

I felt that, due to the timing of events, that, yeah, it was a bit tasteless. Be inspired, write it, but don't release to DVD, before it has played out in the theater, so to speak.

Sorry, I cannot subscribe to your homemade definition of righteousness, morality or ownership of ideas. If I see someone else do a clever trick in a deathmatch, I will instantly copy it. If I find the texturing done by Romero cool I will immitate, and if I see a cool way to code something I will copy and build on top of it.

At no time do I owe any grace period where there is a "cooldown" period where I can't borrow that rocket jump you just discovered, recoil effects on weapons in your mod, or portals in a source port. When Duke Nukem 3D was made there was no obligation to wait until Doom had "finished running in theatres" and I can only object to this way of thinking in the strongest ways. Only in modding competitions can I see a bit of logic in this, during the contest itself, but surely not when it comes to software development.

You know, immitation in the purest form of flattery. If someone copies what you did, smile, and realize you did something someone else thought was truly cool. :)

kb1 said:

Come on, mate, don't hide behind that speech. I talking about moral obligation, not legal obligation.

What does that mean? Just because Eternity was there first I should have refrained from making the already fully working portals in GZDoom also fully working physically?

This is complete nonsense. The only established definition that could be considered a compatible standard is DeHacked and Boom. DECORATE was just as home-grown as EDF, DDF, and a number of others. You mention that to target Eternity, people have to do "double work"? Do you mean ZDoom's format, and some other format? Why is making it use ZDoom's format necessary, if you are targeting Eternity?

Sure, strictly speaking you are right, of course, but I think I understand where he was going: What he was saying is that Eternity, despite having gained quite a bit of features from ZDoom, simply cannot run any ZDoom-packaged map out of the box, even if Hexen map format was made functional. By having chosen entirely different file formats for everything it has effectively isolated itself from this resource and limiting itself to being yet another Boom clone, so to say.

Get this: ZDoom's method is as incompatible and exclusive as anyone else approach that came after Boom and DeHacked. Because, Guess what? Doom never had a monster definition language, except for that which was hard-coded into the executable. ZDoom hacked together their formats just like every other port made their own, since no one was willing to commit to the spirit of collaboration.

Right. But see this in a historic context. Back in 2003/2004 when Decorate came to be, the only serious contender was DDF in EDGE. Which suffered from the *little* problem that the engine was just too different internally to port this stuff to ZDoom. Every other method of defining new actors at that time was a laborious, 1:1 representation of the internal data, including having to manually set up the state linking. So, since there was nothing adequate to collaborate with, I had to start from scratch to make something that satisfied my needs.

@Graf: You have a unique opportunity to end this years-long debate, and bring the people and the ports together. I suggest exposing your plans for ZScript to the community. The entire community. You mention open source. Open up your ideas to the community, and allow all source port developers to mold the language. Drop the "Z" out of the name, and use a name that suggests multi-port compatibility.

Ok, some brief synopsis: The plan for ZScript is to have a language which can be used to write actual extensions to the engine. Not just new actors, but also stuff like new sector movement types or menus.
But here comes the problem with compatibility. This thing completely builds on ZDoom's class system which is such a major cornerstone that porting this to other ports is most likely not possible. It hooks directly into the internal structures and object hierarchy, because that's necessary to use it to add actual class extensions.
The language itself is just a formal grammar. But porting this to other engines is not something that will come easy, I fear. The VM needs a significant amount of infrastructure to work, most importantly an object management system with garbage collection, and obviously a design that the object types that can be created will be represented in such an object system. To my knowledge, the only Doom-based engine aside from ZDoom that did this to a certain degree is Eternity.
ZDoom already had such an infrastructure in place for several years so of course the VM fully builds on that.
But even if that part could be ported, it won't change the fact that the various Doom engines differ greatly in how certain features are implemented, and with significant parts of this having to be exposed to give the language the needed power, it is almost inevitable that large portions of code will not be able to run on other Doom ports with different features or different implementations.
In general, think more along the lines of Lua or AngelScript, but with a dedicated focus on being able to extend natively defined classes (which was the main reason why these languages were deemed insufficient. I would have liked to use AngelScript but it completely falls apart if you want to extend a native class hierarchy.)

Let's come together and make a scripting language/actor definition language that all ports can use, so, finally, there can be some chance of cross-port compatibility. This would redeem and remedy years of vertical non-unified development. Make it generic, requiring a compatibility layer so it can be bolted on to ZDoom, Eternity, Your Mam's Doom, whatever. Doesn't that make more sense than the one man ZShow?

The problem with that goal is that it will always be limited by the lowest common denominator. I'm sorry, but that's really not what we are after here. The point of the scripting language is not just to define new content but also to be able to define new gameplay by virtually overriding some of the engine's core methods in the script. By being able to script engine extensions, it by necessity needs to know a lot about how the engine works and that's inevitably port specific.
A good example for this that's easily understood is ZDoom's inventory system, which is radically different from any other port out there. With ZScript you will be able to program completely new powerup types, but to be able to do that it has to hook into the existing inventory framework consisting of several virtual functions. And that thing is definitely at odds with demo compatibility needs so I'll never see this ported to another engine ever.

Wouldn't it be nice to build a custom monster, and have it appear in Eternity, in ZDoom, and in others?

TBH, for that it might be a better goal to get a DECORATE subset working on those other engines.

Graf Zahl said:

In general, think more along the lines of Lua or AngelScript, but with a dedicated focus on being able to extend natively defined classes (which was the main reason why these languages were deemed insufficient. I would have liked to use AngelScript but it completely falls apart if you want to extend a native class hierarchy.)

I always imagined the "uber-port" - the port in my head that was ahead of even ZDoom - was a port where the entire gameplay loop was done in a scripting language, and instead of endlessly extending a base game, you'd simply modify the base game to your liking. Kinda like Quake 2, but with Lua instead of having to compile a dll file.

Did ZDoom ever consider this route, and if not, why not? I imagine one argument might be that it might be too difficult, but the glut of mods for Quake 2 back in the day makes me think otherwise. I imagine another one would be that it would be too slow for the toughest maps, but isn't this scripting language going to slow things down anyway?

Also, I'm...not quite sure what the heck Garry's Mod does, but did you ever glance at it, and if so what was wrong with its approach?

Graf Zahl said:

TBH, for that it might be a better goal to get a DECORATE subset working on those other engines.

I would LOVE to have a scripting language similar to DDF/Decorate/ZScript implemented into PrBoom+. But, nobody ever said life was a rose garden... ;_;

This marks about the 100th time kb1 has written a long post about how Doom coders should all come together to create a new super-standard.

A few comments from me:

1. EDF was actually started before DECORATE existed. Eternity actually had full actor specification via EDF before ZDoom did. And Eternity also had thing inheritance before ZDoom did. Those ideas were developed independently in ZDoom shortly after (in the case of inheritance I don't think I'd even finished my version before ZDoom started adding it to DECORATE as well, which I thought was kind of cool).

2. Eternity didn't "choose" some of the different formats it supports, it was born with some of them. I'd assume you have to be talking mostly about EMAPINFO. EMAPINFO is just an extension of the SMMU LevelInfo system. It was easier to extend it in place than to ditch it for Hexen MAPINFO. But, if you were up on Eternity history, you'd know EE supports the Hexen subset of the MAPINFO language as well for over two years now. As far as newer developments, they've been made as cross-port compatible as can be done (for example, you can copy-paste Hexen/ZDoom sound sequence definitions directly into an EDF heredoc, because the format is the same).

3. People should let the portals thing go now IMHO, because, first off nothing wrong was done to begin with, and either way, it has worked out unquestionably to both ports' benefit. printz was able to bounce ideas off the ZDoom team while finishing up EE's linked portals, and he couldn't very well have done that w/o ZDoom actually having them also. Porting the stuff into UDMF is a good example.

4. Regarding EE/ZDoom compatibility it might be further along than some of you imagine. Given Hexen map format support being ~99% complete at this point, UDMF fully functional, and DavidPH's ACS interpreter, EE can already run some early ZDoom mods. Not perfectly usually, but they load up and you can make progress, in some cases finish whole levels. It's certainly progress versus two years ago when there were no maps I was aware of that were playable as such.

I'm glad someone said it.

Graf Zahl said:

I am surprised that this comes from an actual programmer who should know his stuff.

I do hope that this was aimed at kb1 and not me. If it was me, I will point out that my post contained no opinions or statements of preferences so none of it should have been surprising.

With that said, since you asked, I do indeed believe that a formal BNF grammar could be made for semicolon-less state definitions. Requiring parens is the easy way, but I believe the four character restriction way could be done by making an action function name (and sprite names) be special token types with mutually exclusive patterns. Yes that would require certain tokens to only be valid within certain rules, and I can't say if Lemon handles that (haven't worked with it directly yet), but this is something that does happen in the C++ grammar (in that case keywords, but effectively the same thing).

Properties also don't require semi-colons. ECWolf in fact accidentally has support for constant expressions in properties, which I didn't really realize I did until Woolie Wool pointed out that random() wasn't being random when used with the sighttime property (because besides damage every other expression would be statically evaluated). In ECWolf's case this is just utilizing the parsing rule for damage expressions so either the argument is a direct constant (which theoretically could be an identifier) or is enclosed in parens. ECWolf does still use the property definition during lexing to determine which form of damage to use, but also to handle integers vs strings (although it is capable of skipping unknown properties and flags). But I can't think of a reason why types couldn't be checked after lexing. ECWolf does of course require comma separated arguments and quoted strings like ZScript.

Now I do see a good argument for not requiring parens to do constant expressions. In which case yes a semicolon is required.

I'd be happy to investigate this if you don't have other objections. More interested in possibly getting rid of the semicolon in states since I think there's a little bit of elegance lost having them there. (I must say the DECORATE state syntax is a really really nice solution to defining state tables in my opinion.) The properties I don't really care about, but I could see it being helpful to transition people from DECORATE.

kb1 said:

1. You mentioned deprecating Decorate. Don't, otherwise all other ports are adding an "old, no-longer-updated" type feature. And, besides, Decorate works. (Fix the oddities, unless in 'compatibility mode'). Fully support Decorate...as a definition language.

I want to emphasize that the "deprecation of DECORATE" comes from the parser being a complete mess of backwards compatibility. The suggestion of using DECORATE as a cross port standard comes from the fact that its core feature set makes very few assumptions of the underlying implementation (it is possible to implement DECORATE as a compile time language for Chocolate Doom if one wanted with probably far fewer compromises than one would think). I don't think this would be possible with ZScript at least not without effectively limiting it to the feature set that DECORATE covers. I know Quasar has a list of reasons that DECORATE is unsuitable as a content definition language for a demo compatible source port. IIRC largely revolving around how vanilla handles pickups and how that interacts with DeHackEd.

In this case the fact that DECORATE is no longer a moving target is of great advantage to others. The oddities don't really need to be fixed on ZDoom's end since well written DECORATE code would work in both a real parser and the messed up one that ZDoom has (see how ECWolf parses DECORATE differently but the dialect looks exactly the same if well written). The only thing the current parser prevents is new language constructs.

So to be clear: If the community adopted DECORATE as a cross port standard and it was useful to add flags, properties, or action specials to facilitate this. I can't see why ZDoom wouldn't support them as well as long as it doesn't require new language features. Another way to think of this is to think about DECORATE vs DeHackEd, or UDMF vs Hexen format. (I'm going to avoid calling those features "deprecated" since it's such a touchy word, but I think the analogy works.)

Also want to remind you that Vavoom implemented DECORATE among some other ZDoom compatibility features, so for awhile it wasn't strictly ZDoom and children.

kb1 said:

Can I, say, bang your girlfriend, while you're still dating her? More on point, can I take your mapset on the day you release it, rename it to 'Check dis out, man', and publish it? Would that be ok with you? Would that fit your 'homemade' set of righteousness? Are you proud of that portrait of your character you painted so vividly? Yikes.

Since when did I say I would do any copyright infringement? There's a difference between copyright and ideas. But okay, I guess I'll just have to be the "bang your girlfriend" ultimate evil in your moral code. I can live with that.

As for taking QZDoom and renaming it - go ahead. Anyone that needs to know already knows where you got it from, so it wouldn't make any difference to me.

kb1 said:

Yeah, I think you could have reached into ZDoom's vast feature set and pulled something else out, and afforded Eternity some time to shine on it's own. The stakes for ZDoom were much lower. Again, maybe I'm wrong, and I shouldn't be speaking for Eternity, but that's how it looked to me...

I decline to comment on this, aside from saying that I am glad that you are not the developer of an active open sourced source port.

NOTHING can run most ZDoom maps except for ZDoom and friends. I know Eternity has been working hard on this, though - bravo! And, nothing but Eternity can play Eternity maps. But Eternity did not choose "different formats" any more than ZDoom chose "different formats". Some people speak as if ZDoom has been self-titled the master of Doom ports, and any port desiring customization must follow ZDoom's conventions, else they are "purposely deviating". Good Lord, help me. It's that bias that drives a stake into all attempts for a standard. Should all port devs should just give up and work on ZDoom? Is that really the goal for ZDoom? Is that why ZDoom adopts the features of all other ports, while adding non-wad-like, non-Doom-like extension after extension to it's own set of features? With ZDoom, the door doesn't seem to open both ways, which, I think unnecessarily sucks. I don't believe there's some evil plot going on, but at times, nothing really prevents that interpretation.

I was just trying to interpret another poster's words. But no matter how you see it, no matter how I see it or even no matter how Quasar sees it, if that's the impression people get from Eternity it's doing some harm to its standing.
I'd really like to see a cross-port standard develop for these things but they'll always be obstructed by the maintainers of more basic port who see no interest in expanding a common standard beyond what it is today.

Don't get me wrong - Decorate is great - the biggest plus being the compactness of the code, and the sheer amount of Decorate code out there. Yes, Decorate can be considered a standard of sorts, by the sheer amount of usage that it has seen. Nothing wrong with that. The ACS action extension mechanism, though, is a bit iffy. But it facilitated a lot of neat things that would be next to impossible otherwise. Credit where it's due, and all.

The ACS extension mechanism is the biggest bootload of trouble. It's something I'd have gladly skipped. Of course mods using that would be too specific anyway, if DECORATE was used as an actual standard, I would never expect it to be all of it with all the cruft, but a cleaned up and streamlined subset of features.

And, I suppose you're right - to allow this language to work in multiple ports would be difficult. It is actually possible, with the addition of a compatibility layer, but that could really slow things down.

The compatibility layer would be the smallest problem. After all the basic functions are mostly the same in most ports so that part should be easily doable. Once things get clear enough to see what is needed, define a set of functions that are to be considered standard and where they are not sufficient, write a few more.

I guess ZDoom has gone too far for a meaningful translation back to Doom's roots. At some point, you have to decide to make a clean break from compatibility if you are to actually get things done in a reasonable amount of time, and that decision has been in the process of being made for many years now. There's no way back. That's not a bad thing, per se, and I suppose my efforts to try to advertise the idea of unity are misguided. ZDoom is now a generic game engine that can emulate idTech1 games, and the other ports are Doom games that can do some ZDoomy things. I can accept that.

ZDoom is far from being a generic engine. It's still idtech1 for a major part, but over the years has made the decision to separate implementation and definition for the most parts. But as things are, it is inevitable that modders want to use the features so making a new scripting language actually have less power than DECORATE would make it an instant dud with modders.

1. You mentioned deprecating Decorate. Don't, otherwise all other ports are adding an "old, no-longer-updated" type feature. And, besides, Decorate works. (Fix the oddities, unless in 'compatibility mode'). Fully support Decorate...as a definition language.

'Deprecation' in this sense means deprecating further feature development. It surely will inherit new functions being defined for ZScript as it works off the same set of resources, but feature suggestions like 'add local variables to DECORATE' will be promptly answered with 'Use ZScript.'

2. Make a clear separation between scripting and Decorate. Decorate is a mobj definition language (with a way to call limited functions with parms, but, mainly a definition language). By making this separation, Decorate can stay Decorate (without semicolons :), without a more complicated parser, etc. ZScript can be script-only, which makes it clean. It can have ways to set states, jump the actor to different states, etc. Take the mobj definition stuff out of it. Otherwise, you'll have to also add weapon definition parsing, menu definition parsing, etc. This means you'll have to support the old way, the new way, etc, and scripting can be it's own thing, and definitions can be their own thing. This also allows your definition files to use all the compactness tricks you desire, without them having to fit into the rigid syntax required by the ZScript parser. Does this make sense?

I won't even consider this suggestion. It serves absolutely nobody to separate stuff that logically belongs together. DECORATE cannot solve one of the biggest complaints modders had: Being able to use symbolic constants and expressions in the actor property definitions. That had been on many people's wishlist but had to be nixed due to the parser's deficiencies.

3. Once your script engine is running, a natural tendency is to convert all the original action functions to script, and running a fully-scripted game. Avoid this tendency. Run the original hard-coded functions, unless the script asks for a custom function. This will keep the frame rate up. Even the best byte-code interpreter will be quite a bit slower than native code. It would be an interesting test to run all the original functions in script mode, but it doesn't really make sense to do it normally.

The core engine is too demanding to be ported, but the top level action functions are another story. Whether a function that calls P_SpawnMissile and S_PlaySound requires 20 or 200 machine instructions to call these functions is utterly irrelevant since they consist of several thousands each to do their work.
I am porting these functions, because they serve as a good test for running real life code to the VM and see how it deals with that.

Porting something like P_CheckPosition or P_CheckSight would be insane, not only for performance issues but also for the ability to debug the code.

Can I, say, bang your girlfriend, while you're still dating her? More on point, can I take your mapset on the day you release it, rename it to 'Check dis out, man', and publish it? Would that be ok with you? Would that fit your 'homemade' set of righteousness? Are you proud of that portrait of your character you painted so vividly? Yikes.

Ouch! That's like comparing apples with oranges. Remember: The source of most Doom ports has been released under licenses that say "You may ... use this code, provided..." In other words: The author has given express permission to use it.
As for copying features, it has been said before that there's really little that Doom port authors can claim to be original ideas. Portals have been done before, scripting languages have been done before, hell, even graphic enhancements have been done before. Guess how my customers would react if I had a store and told them 'Sorry, but I cannot sell that because the store two streets away already does.' That's not how things work and not how they are supposed to work. If you want protection for your ideas you can either not to release the source and not release the specs and build on the laws that have been designed to protect your investment, i.e. copyright and patents. When it comes to open source code, there is one moral obligation I deem important: You have to respect the license the code comes with and give proper credit where you borrowed some ideas and concepts - provided that the source of those is legally clean. And that's the end of the story.

You know, I've modified my view somewhat, partially based on thinking about this thread. If, in my mind, I logically separate the ports into 2 categories: Ports that strive to be generic, robust, editable game engines, and ports that play Doom and friends really good, then I come up with this idea:

Generic game engines can and should concentrate on features and robust scripting, without worrying about cross-port compatibility.

Ports that want to render Doom, Doom II, Heretic, Hexen, Strife, etc, with full accuracy, compatibility, and nostalgia really should have a common mobj definition language, at a bare minimum, at this point. We've all got good Boom and DeHacked support, and we've had that for many years now. We should be able to step up to the next level, and update the 'standard'. Maybe it's Decorate, with all the standard code pointers, and a small handful of generic custom pointers, like "FireMissile", "Melee", etc. Doom is an adult now - it's a bit pathetic that the common denominator is Boom/MBF and DeHacked, I think.

Absolutely and wholeheartedly 'yes!' But be it as it may, it will always be PrBoom+ having the last word here. If its maintainer is not on the boat, there won't be any progress and he's not showing even a hint of it so far. The funny thing is, I believe even PrBoom+ could add a simplified subset of DECORATE without running into serious issues, all it needs to do is output the compiled data in the same form its hardcoded tables currently are.

There's other "bare minimum" things that we should standardize on. What I find to be the worst calamity of customized Doom is the fact that there is no way to take a wad file, and know which port to launch to play it, without reading (and trusting) the damn text file. A launcher has no way to know which port to launch. To me, that is a sad state of affairs. Yeah, I've discussed it before. The threads inevitably turn into pissing contests about insignificant aspects of minutia.

I think we are 18 years too late for that. The damage had already been done with the first release of Boom and sadly every port back then copied the idea of enabling new features by default instead of using a manifest system to selectively enable what a mod needs.
For ZScript we will actually have to consider this seriously. Imagine, ZDoom adds a new actor property 'cakemode', but then runs across an older mod that defined 'cakemode' as a global constant. That mod would break with the new version.

AlexMax said:

Did ZDoom ever consider this route, and if not, why not? I imagine one argument might be that it might be too difficult, but the glut of mods for Quake 2 back in the day makes me think otherwise. I imagine another one would be that it would be too slow for the toughest maps, but isn't this scripting language going to slow things down anyway?

Of course I occasionally have thought about it.
There's just two problems:

1. Performance. You cannot do the physics check in a scripting language which is approx 10x slower than native code and still 3x slower if you do JIT compiling without optimization. You cannot really JIT with optimization because it's too slow.

Porting the high level logic to the scripting side should be doable, but I doubt that we'll fully go there. I am porting some parts to test the implementation with code I know well enough to recognize if something is not correct. Overall, ZDoom is 6MB source code, of which 1MB would be a likely candidate for scriptifying, but not all of it would make sense. My guess is that ultimately half a MB may be converted,most of which is the original games' action functions.



2. Debugging. Finding errors in scripted code is an entirely different thing than debugging native source. It is feasible to do smaller, high level tasks in a scripting language, but if stuff gets too complex you just need some debugging capabilities.

I find it troubling that ports are supposed to run NUTS.WAD (or NUTS3.WAD) smoothly to be deemed valid in terms of performance. Most playable maps are much smaller than NUTS.WAD.

Regarding infringement:

Look guys, as I stated before, it looked/felt like a dick move. It isn't about open source or copyright. And I made that point pretty obvious. Maybe you can't feel empathy - there's a name for that. Maybe you're being intentionally thick. It doesn't really change things. The justifications are irrelevant:

Some random guy on the internet thinks it was a bit dickish. Period. You can learn from that, or write it off, but it's a true observation.

Now if the Eternity devs have turned the other cheek, so be it. Who am I to say differently? I've stated my opinion, which was a valid thing to do based on the OP - a possible insight for why people are "eyeing the exits". It was posted as an insight, not a "make-wrong". What's done is done.

dpJudas: (explains a lot): Your post reads like a punishment to the most generous. You sound like that guy that empties the whole Halloween bowl, disregarding the sign that reads "Please, just take 1 or 2 pieces.". When someone has 1 chocolate glazed doughnut, and 1 plain doughnut, and offers you one, do you always take the better one? Just because you can, does not mean you always should. Anyway, boring.

kb1 said:

2. Make a clear separation between scripting and Decorate. Decorate is a mobj definition language (with a way to call limited functions with parms, but, mainly a definition language). By making this separation, Decorate can stay Decorate (without semicolons :), without a more complicated parser, etc. ZScript can be script-only, which makes it clean. It can have ways to set states, jump the actor to different states, etc. Take the mobj definition stuff out of it. Otherwise, you'll have to also add weapon definition parsing, menu definition parsing, etc. This means you'll have to support the old way, the new way, etc, and scripting can be it's own thing, and definitions can be their own thing. This also allows your definition files to use all the compactness tricks you desire, without them having to fit into the rigid syntax required by the ZScript parser. Does this make sense?

Graf Zahl said:

I won't even consider this suggestion. It serves absolutely nobody to separate stuff that logically belongs together. DECORATE cannot solve one of the biggest complaints modders had: Being able to use symbolic constants and expressions in the actor property definitions. That had been on many people's wishlist but had to be nixed due to the parser's deficiencies.

My mistake - I should have clarified that I was talking about the frames, not the actor properties. Frames should be parsed separately, because of the compression you have accomplished with Decorate, which runs counter to a rigid code parser. Theoretically, you should be able to define frames exactly as you did with Decorate. And, it could be in the same file, with maybe a keyword or symbol that denotes a switch from the code parser to the frame parser, and the reverse. I don't know how you're considering doing this, but I think it would be a mistake to try to parse frames by hacking up your script parser. If you're not doing that, please disregard what I said.

printz said:

I find it troubling that ports are supposed to run NUTS.WAD (or NUTS3.WAD) smoothly to be deemed valid in terms of performance. Most playable maps are much smaller than NUTS.WAD.

I'm with you. But, here's the thing: These tests don't really matter, unless you're trying to play a map, and it's choppy. Then they matter, no matter how fast your port is. Trying to get NUTS.WAD to run well is a milestone, in that regard. Modern ports play smaller maps so fast that, you almost need to throw a NUTS.WAD at your port, as a metric for how fast your port is. In other words, it's a valid metric for raw performance testing/comparison, but I think it's ridiculous to call a port "slow" if it crawls on NUTS.WAD. Having said that, an MBF performance-killing bug was recently discovered and fixed in PrBoom+, using NUTS.WAD! So, it's worth consideration, but not a big deal either way.

kb1 said:

dpJudas: (explains a lot): Your post reads like a punishment to the most generous. You sound like that guy that empties the whole Halloween bowl, disregarding the sign that reads "Please, just take 1 or 2 pieces.". When someone has 1 chocolate glazed doughnut, and 1 plain doughnut, and offers you one, do you always take the better one? Just because you can, does not mean you always should. Anyway, boring.

Is that supposed to be some attempt at guilt tripping for having an opinion about sharing?

Personally I think he's on point, and I agree with him. You and I both know Eternity has more reason to exist than portals. You and I both know portals is not the only thing that Eternity did "first" before the other Doom ports did.

Eternity is not a bad port - but to argue the portals issue like you do implies that it is. That's a worse reflection on Eternity, than it is on ZDoom. Because what's a port worth if that's the only thing it had going for it? Luckily - it's more valuable than some gimmick renderer feature that essentially lets a mapper be lazy and map in layers so that you don't have to use a 3D engine. I know you know that, but I don't think you gave your argument that much consideration.

And in the future, it's hard to take someone seriously when they call port developers "dicky" when they use guilt trip attacks like this.

kb1 said:

stuff about dpJudas

So when are you going to actually read his posts, instead of replying to a completely different conversation and attributing it to him?

I mean I guess we could be talking about an entirely different dpJudas, but if that's the case I don't really get why you're posting this on Doomworld instead of whereever that dpJudas is actually talking.

So, ports should have a common set of new features to define a new standard going beyond vanilla Doom; and at the same time ports should not imitate the new features of other ports because original idea do not steal.

Gez said:

So, ports should have a common set of new features to define a new standard going beyond vanilla Doom; and at the same time ports should not imitate the new features of other ports because original idea do not steal.

That put it far more succinctly than I could have. These two demands are fundamentally incompatible. The moment a port gets a feature and none other can have it, standardization is gone.

Remember, this is a guy who is supposedly developing a port in private, only to be released when it is 'perfect'. Apparently he's too scared that others might 'steal' his code... :D

Sorry, kb1, but you clearly miss the point of open source software by a huge margin.

kb1 said:

I'm with you. But, here's the thing: These tests don't really matter, unless you're trying to play a map, and it's choppy. Then they matter, no matter how fast your port is. Trying to get NUTS.WAD to run well is a milestone, in that regard. Modern ports play smaller maps so fast that, you almost need to throw a NUTS.WAD at your port, as a metric for how fast your port is. In other words, it's a valid metric for raw performance testing/comparison, but I think it's ridiculous to call a port "slow" if it crawls on NUTS.WAD. Having said that, an MBF performance-killing bug was recently discovered and fixed in PrBoom+, using NUTS.WAD! So, it's worth consideration, but not a big deal either way.

The thing is just: NUTS is not a reasonable test case. It's a map that will inevitably not test the code's efficiency but the code's simplicity. In order to make this work well the only option is to reduce code, reduce more and more, read: stripping away features.