Defining a new enhanced source port standard (Boom+/MBF+)

 

19 minutes ago, Da Werecat said:

When it comes to the automap, MBF has a solution: the OPTIONS lump. Eternity supports it. The problem is: PrBoom+ stubbornly refuses to acknowledge its existence for whatever reason. Then there are palette swaps - a minor inconvenience, but still.

 

If the OPTIONS lump is so unloved, then maybe there can be another solution. First thing to decide: would the new values always override the user's settings (like OPTIONS), or are they just the defaults that take effect once and then can be changed by the user? In the latter case: where to store the new values? We don't want to replace the settings that are good for the default palette in the config.

 

The OPTIONS lump is a dead end. Even if my PrBoom+ fork turns out into a separate port, this will be one definitive no-go.

 

You cannot make an engine specific configuration file part of any standard. ZDoom couldn't use the vast majority of its content, not to mention that much of it has no place in a mod configuration. It's par for course for MBF which, despite some interesting features suffers from some of the most hackish and short sighted feature implementations ever. Strange that Eternity keeps this around, it would have been the first thing I'd have disabled.

 

 

Share this post


Link to post

For hardcoded palette colors, you could imagine a COLORDEF lump.

 

What would go in there? Automap colors, and translation start/stop indices. Anything else?

Share this post


Link to post

I think palette swaps can be externalized with color tables. Font recolors are already done this way, AFAIK.

Share this post


Link to post

 

@Graf Zahl For these custom lumps, I can imagine I would just adapt the new idea, for example: (COLORDEF) to be parsed to a DDFCOLM DDF (sort of like how our Dehacked plugin works)..unless I have the wrong idea here.

 

I see lots of mention of ACS in the UDMF spec... I'm assuming the 3DGE equivalent, RTS, is out of the question. To be compatible with the others, would we be "forced" to write an ACS parser? Just wondering the best way to handle this. .

Share this post


Link to post
59 minutes ago, Coraline said:

 

@Graf Zahl For these custom lumps, I can imagine I would just adapt the new idea, for example: (COLORDEF) to be parsed to a DDFCOLM DDF (sort of like how our Dehacked plugin works)..unless I have the wrong idea here.

 

I see lots of mention of ACS in the UDMF spec... I'm assuming the 3DGE equivalent, RTS, is out of the question. To be compatible with the others, would we be "forced" to write an ACS parser? Just wondering the best way to handle this. .

ACS isn't part of UDMF itself and I don't think it's being seriously considered as part of the phase 1 plans here, though a shared scripting standard would be pretty cool to have.

Share this post


Link to post
1 hour ago, Coraline said:

 

@Graf Zahl For these custom lumps, I can imagine I would just adapt the new idea, for example: (COLORDEF) to be parsed to a DDFCOLM DDF (sort of like how our Dehacked plugin works)..unless I have the wrong idea here.

 

For every new lump there will be a reference parser. Normally you should just be able to take that parser and redirect its output to where you need it, if you already support the feature it implements.

 

 

1 hour ago, Coraline said:

 

I see lots of mention of ACS in the UDMF spec... I'm assuming the 3DGE equivalent, RTS, is out of the question. To be compatible with the others, would we be "forced" to write an ACS parser? Just wondering the best way to handle this. .

 

ACS is only relevant for ZDoom and ZDEE namespaces. There's some mention because these namespaces need some traits defined how ACS interacts with UDMF's properties. The base 'Doom' namespace does not contain any ACS and would not even be able to use it.

 

If we wanted to add a scripting language, what would be other options? RTS? FraggleScript? I could port FraggleScript to PrBoom, but I'd take out string variables because they cannot be cleanly implemented without a working garbage collector - but that'd limit the feature set quite dramatically. I'm also not sold on the language. It's a relic from the past better left shrouded in obscurity. And about RTS I simply do not know enough to even decide if it's a viable option.

 

Share this post


Link to post

The Automap colours are trivial. My port looks up the colour rgb in the standard palette and then finds the closest match to that rgb in the changed palette in the pwad.

 

Share this post


Link to post

ZDoom is doing the same thing when rendering the automap on a paletted display.

 

Share this post


Link to post

It is not surprising that there is another animated switch idea.  I could accept some alternative way to do the same thing.

Could the people that propose using such actually write up something so the rest of us can judge its merits too.

Is this going to be overkill like the DECORATE thing.

 

Please allow me to point out one counter-arguement, without someone jumping down my throat.

At least my idea only extended the existing naming convention, and did not require any lump writing to use it or any lump parsing to implement.  Why does every put down of one of my ideas have to automatically assume that a two-ton sledge hammer version in some other port or format is always the way to go.  My ideas are claimed to be way to complicated for this phase, but the alternatives proposed are way more complicated.  I could implement one of my ideas in a weekend.  I have no idea when I could actually implement these overly complicated lump formats.

 

What good does it to say that Hexen has this covered, if we are not using Hexen.

 

I know that DoomLegacy has some switch definition lump handling, but only adding switch definitions.

I think it is a SWITCHES lump but I am on the wrong machine to check right now.

It is hard to find or know what is in there until know what I am looking for.   I do not remember any animated switches.

 

Share this post


Link to post
On 5/25/2017 at 3:55 AM, Graf Zahl said:

 

I don't think there's any need to bother with the hard coded section at all. The first thing any port should do here is to export the internal definitions so that there's one less concern. IIRC even Boom moved these tables to an ANIMATED and SWITCHES lump

 

One thing should be clear here:

ZDoom's ANIMDEFS better die. It is based on the original Hexen format and lacks sufficient structure to be used as a universal definition lump. If we go at it my suggestion would be to use something based on UMAPINFO syntax. I think for everything (except DECOLITE which has different needs) we should stick to that if possible.

But please let's not to a 'one size fits all' format like EDF, DDF or DED. These inevitably carry too much baggage around. If the various things are properly separated it will be a lot easier to support them because you can do one without the other.

 

I like the idea of using the same parser for most all text lump needs (I understand that DECORATE/LITE needs something more). On the "one size fits all" thing, let me be more clear:

So you want to use the same parser, right? If we simply require a header line:

type UMAPINFO
{

}

type ANIMDEFS
{

}

The data could *optionally* live in one lump, or in separate lumps, and the parser could handle either one. I suppose, if using single lumps, the lump name could serve as the header line. The data within each type can be formatted however necessary - no baggage. If the type exists, it gets parsed, otherwise not. It's of course easy to support both: A lump, called CONFIG or something, would look like the code box above. The one lump per type version would omit the header, and use the lump name to denote type.

 

Treat this as just a suggestion, for maximum flexibility. This is the route I took with KBDoom, and it's worked out pretty good so far. What's nice is that, with one lump you can configure the whole game (for the most part). Cause, otherwise, we need to come up with different, unique names for all these lumps - all the good names have been taken! Please consider it. It's a logical result of using one parser, so it makes sense to me. It's up to you, though - I'll roll with whatever you think is best.

 

On 5/25/2017 at 7:43 AM, Da Werecat said:

I have a proposition that could benefit the artsy types.

 

Right now anyone who's trying to make a new palette from scratch is faced with certain challenges. A lot of the colors are reserved by the engine for certain things, namely palette swaps and the automap. They need to be accounted for when creating the palette layout. Now, while the palette swap ranges are pretty easy to work with, automap colors are a giant mess, especially considering that different ports have different default values for them.

 

One way to deal with this right now is to make a sensible, convenient palette layout, develop the graphics, and then mangle the palette so that it works with the default values of the target engines. One drawback is that once the palette (and all the related lumps) is mangled, you'd better be sure that it's finalized, because otherwise you'd either have to work on the mangled version (major PITA) or re-mangle it again.

 

Another drawback is that any user who has custom automap colors will have to go and change them. This problem can only be solved by making the layout exactly like Doom, preferably with similar hues in their respective ranges. Yuck.

 

When it comes to the automap, MBF has a solution: the OPTIONS lump. Eternity supports it. The problem is: PrBoom+ stubbornly refuses to acknowledge its existence for whatever reason. Then there are palette swaps - a minor inconvenience, but still.

 

If the OPTIONS lump is so unloved, then maybe there can be another solution. First thing to decide: would the new values always override the user's settings (like OPTIONS), or are they just the defaults that take effect once and then can be changed by the user? In the latter case: where to store the new values? We don't want to replace the settings that are good for the default palette in the config.

 

User settings are one of the most unique things in ports. Trying to dictate what settings a port has is against the philosophy of this standard. And, what you describe would make that a necessity. The OPTIONS lump cannot be made generic - it will always be a port-specific thing. I guess what you meant was being able to change the automap colors to match your new palette scheme by using the OPTIONS lump. Personally, I find the idea of a WAD changing my user settings a bit repulsive, in most all cases, with a few exceptions. I could see a WAD dictating if jumping is allowed or not, for example. But I believe that could be done in a more straight-forward way, like a MAPINFO lump, for example.

 

Regarding the palette, it is already user-configurable in a standard way. I do agree that tools to make it easier to work with would be nice, and I recognize that changing the palette affects a lot of things: The menu, for one. Also the automap. But both of those things are also extremely port-specific. This is something that requires a huge amount of planning, and careful execution. I am open to suggestions, but I must say that I am not comfortable trying to squeeze a palette remapping scheme into a Phase 1 spec. There's just too many things to research, and such a change would be difficult to gain universal acceptance for. It raises to many concerns: How would it work with true-color rendering? Custom colormaps? Translucency? Keeping the map and menu readable? Split-screen faked palette flashes? Real palette flashes? Ports that recalculate palette values? To make this universal requires an understanding of how *every* port's renderer works.

On 5/25/2017 at 0:01 PM, Coraline said:

 

@Graf Zahl For these custom lumps, I can imagine I would just adapt the new idea, for example: (COLORDEF) to be parsed to a DDFCOLM DDF (sort of like how our Dehacked plugin works)..unless I have the wrong idea here.

 

I see lots of mention of ACS in the UDMF spec... I'm assuming the 3DGE equivalent, RTS, is out of the question. To be compatible with the others, would we be "forced" to write an ACS parser? Just wondering the best way to handle this. .

Scripting is *way* down the line for this spec. Every port dev that wants to do scripting has their own ideas for what would be the ultimate script engine. I have my own plans. But ACS is actually a standard, at least the original Hexen version. To render Hexen, you need an ACS VM (or the ACS needs to be converted at runtime to your favorite script engine).

On 5/25/2017 at 1:22 PM, Graf Zahl said:

 

For every new lump there will be a reference parser. Normally you should just be able to take that parser and redirect its output to where you need it, if you already support the feature it implements.

 

ACS is only relevant for ZDoom and ZDEE namespaces. There's some mention because these namespaces need some traits defined how ACS interacts with UDMF's properties. The base 'Doom' namespace does not contain any ACS and would not even be able to use it.

 

If we wanted to add a scripting language, what would be other options? RTS? FraggleScript? I could port FraggleScript to PrBoom, but I'd take out string variables because they cannot be cleanly implemented without a working garbage collector - but that'd limit the feature set quite dramatically. I'm also not sold on the language. It's a relic from the past better left shrouded in obscurity. And about RTS I simply do not know enough to even decide if it's a viable option.

 

@Graf Zahl Why would you switch away from ZScript? From what I read, it seemed to be rather fully featured. I must say that, to me, it seemed like it requires some pretty advanced programming skills to master, but, at the same time, extremely powerful. Just out of curiosity, did you build it from scratch, or was it a package that you tailored for Doom's specific needs? Either way, it's quite impressive.

 

I believe that the choice of script engine will be the most difficult thing for us to agree on. I've been slowly working on my own language that resembles BASIC with some C syntax for convenience. Eventually it will expose an event model and a set of pre-defined objects. I currently have it working in runtime, interpreted mode, so it not real fast, but it does work. The next step is to modify the interpreter to output byte-code, and to build a VM to execute the code. Eventually I'm going to try to get it to output x86/x64 assembly to build .EXEs and .DLLs, but that's a long ways away, and it's not needed for Doom. A byte-code compiler/execution engine is plenty fast for Doom, and cross-OS compatible.

 

My point is that fully-powerful scripting that has access to Doom's internal objects requires radical changes *everywhere* in a port, and I don't believe we will ever be able to define a spec that all ports will welcome. I could be wrong, though. I would like to study RTS and ZScript in detail though. I also want to see what Eternity is doing, but last time I looked, I think Quasar was switching to a new scripting engine, right?

4 hours ago, wesleyjohnson said:

It is not surprising that there is another animated switch idea.  I could accept some alternative way to do the same thing.

Could the people that propose using such actually write up something so the rest of us can judge its merits too.

Is this going to be overkill like the DECORATE thing.

 

Please allow me to point out one counter-arguement, without someone jumping down my throat.

At least my idea only extended the existing naming convention, and did not require any lump writing to use it or any lump parsing to implement.  Why does every put down of one of my ideas have to automatically assume that a two-ton sledge hammer version in some other port or format is always the way to go.  My ideas are claimed to be way to complicated for this phase, but the alternatives proposed are way more complicated.  I could implement one of my ideas in a weekend.  I have no idea when I could actually implement these overly complicated lump formats.

 

What good does it to say that Hexen has this covered, if we are not using Hexen.

 

I know that DoomLegacy has some switch definition lump handling, but only adding switch definitions.

I think it is a SWITCHES lump but I am on the wrong machine to check right now.

It is hard to find or know what is in there until know what I am looking for.   I do not remember any animated switches.

 

@wesleyjohnsonI can't speak for anyone else, but the only thing making me want to "jump down your throat" is that your taking the stance that your ideas have been rejected - they haven't. I have not put your ideas "down", I've put them into a spreadsheet, along with all of the other ideas in this 13 page thread, as well as the other 6-page thread, and the UMAPINFO threads, the UDMF threads, and a few ideas of my own.

 

You are brainstorming on ways to get Legacy to do what you want, and you are sharing those ideas. This is good. But, please note that I am developing a spec that must not only work well in other enhanced Doom ports, it must also work in ports that support Heretic, Hexen, and Strife. That's the ideal type of solution. I mentioned Hexen, because, yes, it does manage animated switches, so there is a precedent. To be universally accepted, any animated switch spec had just better be able to seamlessly handle animated Doom switches, animated Heretic, Hexen, and Strife switches.

 

Now, you may never want to add support for those other games in your port, and that's your prerogative, which will be respected by the spec. It is commendable that you've found a way to get animated switches to work without a configuration lump - that's a nice feature. But some other ports cannot use this method to satisfy all requirements to authentically, accurately render each game they support.

 

For example, in some games, each switch can have a different sound. Your method does not allow per-switch sounds, right? Requiring the config lump also requires ports to add fields to their structures to support arbitrary, unlimited creation of animated switches with custom sounds, custom timing. But, best of all, the config lump lets ports exactly emulate Doom, Heretic, Hexen, and Strife switches completely, without needing any more solutions - everything is covered.

 

Yes, it is a bit more complicated to have to get the info from a lump. But that parser is part of the spec, and it will be used for ALL text config needs, so it's a one-time cost, that provides access to animated switches, UMAPINFO, animated textures, custom sound definitions, ambient sounds, terrain definitions (for splashes), and everything else I can't think of right now.

 

The idea is that you keep your core engine logic clean of special-case conditional logic, because, instead, everything is clearly defined in the config lump - you simply follow the parsed data as-is. It removes all restrictions on, say, lump naming schemes, hard-coded timings, etc. Everything is user-defined, which is what allows all ports to take advantage of the configuration. The default configuration allows ports, like Hexen ports, to render the switches in a demo-sync-compatible way.

 

None of this prevents you from adding your animated switch scheme to your port. But, if you later accept the spec, you now have to handle 2 ways to define custom switches.

 

Yes, once it is complete, the first rough-draft will be visible to all, in a new thread, available to be judged, picked apart, modified, tailored, and tweaked to perfection. But, this is going to take time: As I have mentioned before, this is a huge project. I am logging, sorting, and cleaning up all the viable ideas in these threads. I have downloaded the Doom, ZDoom, and Eternity Wikis, and studying them. I have downloaded the latest source of all the ports I can find. I am doing my best to study the source of all these ports. I am discussing ideas with developers. I am trying to boil all of this information down into an inviting, approachable spec that takes each ports innovations into account, avoids stepping on anyone's toes, avoids conflicts, repairs inevitable conflicts, etc.

 

Furthermore, I will be implementing the first spec into my version of PrBoom+, to verify its correctness, and to provide a reference implementation. This reference implementation must maintain demo sync, both for old and new demos, and must preserve savegame compatibility. Once that is done and verified, I can confidently release the first rough draft of the spec, complete with fully-commented, working reference source, with key markers in the source making the spec-specific changes easy to find.

 

Once again, your proposal is quite clever, and a good solution for some ports, but I am trying to reach a bigger audience, and provide a solution that works for any advanced port, the same way in each port. That justifies building the "two-ton sledgehammer" once, and then calling *everything* a nail.

 

It's worth mentioning that, yes, ZDoom has excelled in capturing the needs of all of the idTech1 games, with a lot of their config structures. In other words, their methods have been proven to be solid, but some could claim that they were built hastily. We have an opportunity to build a clean method to handle all of the data needed by advanced ports. I don't just want you to implement the spec - I want you to want to implement it, because of the good things it will bring. The fact that you mention animated switches invokes me to realize that we need a universal way to handle them. I appreciate that you've done this. And the same appreciation goes to the linedef discussions, the "Doom2017" plans, etc. The spec will try to implement some of these ideas, if they can easily be supported universally.

 

My hope is that, after implementing the spec, some ports will gain a lot of functionality. And other ports will be able to load those same maps that use that new functionality, even if they already supported those functions. I want you on board with this - I'm not trying to bash your ideas - quite the opposite. Some of the resistance is that your ideas need to be expanded to be sufficient enough to supply the needs for all ports. Do you understand?

Share this post


Link to post
11 minutes ago, kb1 said:

Why would you switch away from ZScript? From what I read, it seemed to be rather fully featured.

For one, it's heavily built in a way that's based around how ZDoom handles things. But mostly, it's just entirely out of scope for a Boom+ thing.

 

The talk of scripting for this standard is for map-based scripting, ala what Hexen ACS is and what ZDoom ACS used to be before it got extended into oblivion. ZScript, on the other hand, is mostly based around everything else besides the particulars of each map. I mean, you can ( or will be able to? Haven't kept up with whether or not it's implemented or not ) create new thinkers - as in, the things that operate doors and floor movements and such - for use in maps, but when you script in ZScript, it's for the entire game. It allows for whole new action functions for actors, lets you hook into various game events to create your own effects when they occur, etc..

 

Basically, something I'm pretty sure most ports that stick around Boom-compat would have no interest in implementing and maintaining, since it's by far one of the best ways to make something completely un-Doom in Doom. Since the goals with this is just to have better capabilities to expand Doom, it's pretty inappropriate.

Share this post


Link to post
2 hours ago, Arctangent said:

For one, it's heavily built in a way that's based around how ZDoom handles things. But mostly, it's just entirely out of scope for a Boom+ thing.

 

The talk of scripting for this standard is for map-based scripting, ala what Hexen ACS is and what ZDoom ACS used to be before it got extended into oblivion. ZScript, on the other hand, is mostly based around everything else besides the particulars of each map. I mean, you can ( or will be able to? Haven't kept up with whether or not it's implemented or not ) create new thinkers - as in, the things that operate doors and floor movements and such - for use in maps, but when you script in ZScript, it's for the entire game. It allows for whole new action functions for actors, lets you hook into various game events to create your own effects when they occur, etc..

 

Basically, something I'm pretty sure most ports that stick around Boom-compat would have no interest in implementing and maintaining, since it's by far one of the best ways to make something completely un-Doom in Doom. Since the goals with this is just to have better capabilities to expand Doom, it's pretty inappropriate.

I think I misinterpreted what Graf was saying, and furthermore, I wasn't clear with my question. I know there's a "Z" in ZScript. I guess I was just surprised that Graf would consider adding yet-another script engine, for the sake of cross-port support. Naturally, I love the idea, but getting everyone to accept it seems like a pipe dream.

 

On the idea of FraggleScript, from what I read, it seems like a pretty sweet language. I would definitely implement string support. There's a few schemes to minimize garbage collection:

  • Literal constant strings can stay where they are
  • Fixed-length strings could be implemented, avoiding GC altogether.
  • I like allocating blocks of, say 256 bytes for strings. That allows a lot of concatenation of the sizes most likely used in a Doom script: "Player " & X & " did something..." will totally fit in that block, and allow plenty of extra manipulation. If your string grows beyond 256, allocate another block. It typically wastes a small amount of memory, but the GC becomes shit-easy: You don't even need to move the strings - just mark deleted string's blocks empty and available. It's also faster when concatenating if done that way.
  • Defer GC till the end of the map (unless you're running out of memory, which might suggest too much string manipulation by a crazy set of scripts), avoiding the delay

But, again, one of the joys of building an engine is in building a script engine to control it - it's something I've been thinking about for a long time. I imagine other devs feel the same. Regardless, however, a standard script language would be sweet...but, far away.

 

EDIT: On second thought: Here's the problem with a universal script language (USL): All ports are different. In KBDoom, I have a thing flag STOP_BOUNCING_IN_WATER, or something (I can't remember right now). ZDoom may have a similar flag, or it may not. If so, it's almost definitely named differently. So, USL must only be able to manipulate the original flags. If you add a port-specific extension to get to the other flags, BAM! your script is incompatible.

 

So, with USL, you're left with simple stuff: spawning a monster when crossing a line, triggering a door to open when a BOSS dies, playing sounds after some universal condition happens (like a monster death, line trigger, or so). Basically original ACS. You end up with a language watered down so much that it's not far ahead of voodoo doll scripting. A Doom script language should be able to utilize all the port's neat features, but a universal script just can't.

 

Short of adding all features to all ports, I don't have a solution to this problem - unless we go for the simplistic language. It's too involved and far down the road for me to consider it just now. I am getting a bit overwhelmed, and need to focus on the first phase.

Share this post


Link to post

Why not use Fragglescript as a base, change it up to suit things better and give it a different name to separate it?

Share this post


Link to post
9 hours ago, kb1 said:

Personally, I find the idea of a WAD changing my user settings a bit repulsive, in most all cases, with a few exceptions. I could see a WAD dictating if jumping is allowed or not, for example. But I believe that could be done in a more straight-forward way, like a MAPINFO lump, for example.

Providing new automap colors serves the same purpose: preventing the problems that the user's settings might cause with the wad's content.

 

MAPINFO works. Anything would work, even the automatic translation mentioned earlier, although it might make the color-picking menu implementation less straightforward.

 

9 hours ago, kb1 said:

How would it work with true-color rendering?

What are the potential problems here?

 

9 hours ago, kb1 said:

Custom colormaps?

Should be provided by whoever made the wad with an altered palette. As is the translucency table, if the port in question requires it.

 

9 hours ago, kb1 said:

Keeping the map and menu readable?

Graphical assets should also be provided.

 

9 hours ago, kb1 said:

Split-screen faked palette flashes? Real palette flashes?

Palette flashes are contained in the palette lump. And if the port ignores them, doing them in real time instead, then what's the problem?

 

9 hours ago, kb1 said:

Ports that recalculate palette values?

Not sure what that even means and why would anyone want to do that.

 

It seems that most (if not all) of your concerns are aimed at the very idea of using modified palettes. But it's already happening - more and more with time, as the tools are getting better and the people are getting tired of the default palette's limitations.

 

Pretty much all that I'm asking for is to allow providing sensible automap defaults with the wad, instead of forcing the user to go to the menu and remap all the colors manually (screwing the previous settings if they forgot to back up the config).

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now