ID24 - a new feature set standard

[GooberMan]

So. Surprise! id Software licenced my rendering code from Rum and Raisin Doom, and also Boom support that I was planning on writing. The new episode that was being developed though needed MBF21 support, so I didn’t stop at Boom. And I didn’t stop at MBF21 either. There were some features we needed as a commercial product that didn’t exist as accepted community standards supported across a wide range of source ports. So what started out as one thing turned into a new feature suite that we are calling ID24. It is a superset of MBF21 and DSDHACKED, meaning that we also support the Boom and MBF standards.

 

Most of the code exists in Rum and Raisin Doom right now. But not all of it. Xaser filled in a number of cracks and features that I didn’t have time to implement/bug fix/just plain do properly in the first place, and as such R&R actually isn’t ID24 compliant right now - and in fact is called RNR24 in there. Everything required for full compliance will be making it back into the Rum and Raisin Github in the near future (ie when I have the time).

 

We also didn’t get to finish everything before we had to upload our builds to consoles. So as it stands right now, even Doom + Doom II is not fully ID24 compliant. Check my github logs, you can see how close to the wire I was at getting things done (and full disclosure, this is not my day job and thanks to needing to travel to Australia for a couple of months for said job I didn’t have as much time as I would have liked to get everything in order with time to spare). My intention however is for Rum and Raisin to be ID24 1.0.0 compliant at a minimum.

 

Anyway, before we get to the meat, here’s a video. I like videos. This is actually a wad that I have on Github. It’s a quick showcase of a bunch of features (and Boom oddities) that I’ve been throwing together.

 

 

In many ways, this is a conservative update. We stuck mainly to what was required by what we were shipping in a number of cases. But as you can see in the video, there’s a few things that went into the standard because it works in Rum and Raisin and thus it works in ID24. And certainly in terms of scope, it’s not exactly conservative at all.

 

The shortlist of new features:

 

• A standard for JSON to be used as a data format
• Customisable demo loops
• Customisable finales (including continuing to another map)
• A game configuration lump
• New linedef types
• New thing types from Legacy of Rust
• Customisable animated backgrounds for the finished/entering screens
• Customisable status bars (including full screen bars)
• Customisable skies, including PSX Doom/Doom 64’s fire skies
• Custom translation lumps
• And then there’s the DeHackEd capabilities
  • Limits removed on weapon and ammo types (ie you can define entirely new ones)
  • Nightmare/respawning monsters variables exposed
  • Item pickups definable with parameters instead of hardcoded to sprites
  • Transparency maps assignable to any frame
  • Translation lumps assignable to any thing
  • Custom strings
  • Negative index support

 

That last point definitely needs some explaining. DSDHACKED put us in a slight pickle. With the entire positive index range available to users, we couldn’t make assumptions about which indices were and weren’t used. Which, well, when id Software and its affiliates want to add to the game and support the community’s decades of work, it required a bit of code jiggling. As such, the full 2^32 index range is available - but with very specific limitations on what negative indices can be used by who and for what. Short of throwing DeHackEd out for something else, this gives everyone the room they need to grow the game.

 

Also full disclosure, you don’t get demo compatibility with this code outside of vanilla Doom. The code I wrote is a clean room reconstruction, using documentation and secondary resources only. And as demos are explicitly reliant on implementation (and its quirks) I’d have to go through disassembly to do a full reverse engineering pass to get it right. For legal reasons and time constraints, it has to be done like that.

 

We’re also missing a couple of things from MBF - namely, dogs and friendlies. The bouncing code is very likely wrong too (although the original implementation has been described to me solely with varying obscenities by everyone familiar with it, so I think I’m just happy that things bounce at all).

 

Which brings me to my big point - documentation. The documentation we’ve used over the years really isn’t complete enough to do a clean room implementation. I’m constantly bug fixing my code as we find a mapset that expects things to operate in a certain way. So for ID24, I decided that I’m going to lead by example. The specification documents for ID24 are very detailed, and take the path of being explicit rather than leaving things to be implied or assumed to just be the case because that’s how previous code works. It is my intention that any port implementing these features can follow this guide without looking at my code in Rum and Raisin and still come up with something that’s 100% compatible.

 

The ID24 0.9.1 spec is live on a Google drive.

 

We know this feature set is a lot to take in, and that source ports will need a bit of time to catch up. But we do need the work done here as a platform for the future.

 

But short story, we've got a feature set here that allows the community's work over the decades to be exposed to a wider audience via the console releases, adds new features to increase the modding capabilities of the game, and leave a path open for both id Software and the community to expand the game without stepping on each other's toes. We hope y'all like what we've got here.

[esselfortium]

This is exciting stuff! Thank you for your hard work.

[floatRand]

Great job. Congratulations on getting your renderer licensed over.

 

[JackDBS]

Super fucking happy you made a map to show off the features just like the BoomEdit map. Can’t wait to have fun with these!

[GooberMan]

Also check the GitHub logs since I've both extensively commented and had to fix my many fuck ups along the way.

 

The Boom support in particular is very visible in the p_genericactions.cpp and p_lineactions.cpp/h files.

 

This is far from my first rodeo on clean room engineering. My professional credits include Game Room for the X360, an emulator service published by Microsoft under the same "clean room" restrictions. It's one of the reasons in fact that I was very confident I could do it - when you hear C64 music running through emulator code you wrote on a console after using documentation only, a source port is far less scary in comparison.

 

Except for Lee Killough's code, only Satan knows what is going on in there. Using my current methods, I don't ever expect to match what he wrote for MBF.

 

[Terrcraft]

does full id24 require replaced plasma and bfg for the incinerator and calamity blade to work? or is that not the case? 

[GooberMan]

10 minutes ago, Terrcraft said:

does full id24 require replaced plasma and bfg for the incinerator and calamity blade to work? or is that not the case? 

 

ID24 makes them standalone weapons and ammo types that do not replace the plasma gun/BFG.

[Terrcraft]

Just now, GooberMan said:

 

ID24 makes them standalone weapons and ammo types that do not replace the plasma gun/BFG.

Thanks. I thought because of the implementation in legacy of rust this wasn't the case.

Its crazy that we have new official doom enemies and weapons.

[Bauul]

Congrats on the release!  As someone who delved deep into MBF21 when working on Evi 2 it's exciting to see some of the limitations I ran into (such as being able to add new weapons!) being addressed.

 

Is there anything in ID24 that takes a fundamentally different approach to MBF21 (or earlier), or is it all iterative improvements?

[GooberMan]

3 minutes ago, Bauul said:

Is there anything in ID24 that takes a fundamentally different approach to MBF21 (or earlier), or is it all iterative improvements?

 

Iterative for a good number of things. I straight up wasn't going to do new lump types without a standard format like JSON though, so I spent quite I bit of effort on making sure that was well defined as a standard for both tools and the game code to handle.

[LuciferSam86]

Amazing stuff! So in the future R&R will have an open source GPL implementation of id24 so source ports devs can take inspiration? I mean except for the assets which are under a commercial license ( like the iwads) .

 

 

[Redneckerz]

Having taken a peek glance and seeing some of the video, this is awesome! Its great that a universal format like JSON is used for this in a clever way that indeed brings community work closer to consoles without interfering with NDA's.

 

The video helps highlight what the RNR24 and ID24 spec is capable of, so this is a great future up ahead with this official id-extension. Hell yeah.

[GooberMan]

3 minutes ago, LuciferSam86 said:

So in the future R&R will have an open source GPL implementation of id24 so source ports devs can take inspiration

 

Specifically a complete implementation. Features were developed in the open in Rum and Raisin before merging in to the official port. Fixes and changes were made from there and are due to make their way back to Rum and Raisin in the near future.

[Trov]

Are there complevel/GAMECONF executable/etc traits which weren't implemented in time for Nightdive Doom release?

 

Disabled lost soul limit is implied by boom's -cl9 but in Nightdive Doom the lost soul limit is still in place when used with a GAMECONF specifying "complevel9" executable type.

 

I can't figure out how to get a WAD to disable the lost soul limit in Nightdive Doom. With a GAMECONF with executable set to "complevel9" or with "comp_pain 0" in the options field lost souls are still limited.

 

I've read others having trouble disabling "comp_falloff" as well to stop monsters from being pushed over ledges.

 

 

Same question for UMAPINFO - one thing not working as expected is UMAPINFO's "interbackdrop" field in Nightdive Doom does not seem to support a fullscreen graphic (such as INTERPIC) and displays only a black screen, and only flats seem to work properly. UMAPINFO's spec states that it should support a fullscreen graphic too.

 

 

Main reason I ask is I am wondering if these behaviors (forced lost soul limit for complevel 9; only flats allowed for "interbackdrop") are considered to be part of the ID24 spec or if these are slated to be fixed in Nightdive Doom eventually.

 

 

Otherwise, as for the ID24 spec itself, I think one easy cool addition for finales to make them a little bit even less hardcoded would be the ability to specify vertical bunny scrollers that scroll up, or horizontal ones that scroll towards the left, as well as setting the tic delay before it starts scrolling, and how fast/far to scroll. This would enable Star Wars-like or Marathon-like scrolling story screens where you can read text elements as they come in.

Edited August 10 by Trov

[Xaser]

5 minutes ago, DeafPixel said:

Very concerned about this. I don't think you can make a standard that is simultaneously good for the Doom modding community and that "ensures the future commercial viability of Doom and Doom II". I understand that there was a lot of work put into this, and while I would love to congratulate that work, I think its slightly scummy to make it dependent on owning Bethesda's proprietary port. This seems to me like a walled-garden approach, and I am not enthusiastic if this is meant to be part of the future of the Doom modding community.

 

The purpose of this spec is to make it not a walled garden, i.e. explicitly document these features so they're available for community ports. Also see Gooberman's post above -- the cleanroom reimplementation is open source (from Rum & Raisin); id/Bethesda licensed it from Gooberman for use in the official port. Same process as when fraggle's DEHACKED code from Chocolate Doom was licensed for use in the Unity port a while back.

 

1 minute ago, NightFright said:

External ports will require access to id24res.wad for ID24 compatibility? I mean OK, it's probably similar to requiring an IWAD to function in general, but... a file to enable a standard to function? Seems a bit odd.

 

The standard includes definitions for the new expansion's monsters and weapons (taking all new slots, not replacing any old ones); the resource contains the sprites and sounds for these. Though nothing's stopping the community from making a Freedoom-esque libre resource either (that'd be rad).

[LuciferSam86]

32 minutes ago, Xaser said:

 

 

The standard includes definitions for the new expansion's monsters and weapons (taking all new slots, not replacing any old ones); the resource contains the sprites and sounds for these. Though nothing's stopping the community from making a Freedoom-esque libre resource either (that'd be rad).

I suppose behind the res file there's the decohack definition right ? But if someone use a clean room approach to rewrite the decohack definition and there aren't any patents on the file, there's no problem? That's actually cool.

 

And yeah I think Gez got the idea correctly:)

[plums]

9 minutes ago, LuciferSam86 said:

I suppose behind the res file there's the decohack definition right ? But if someone use a clean room approach to rewrite the decohack definition and there aren't any patents on the file, there's no problem? That's actually cool.

 

And yeah I think Gez got the idea correctly:)

Xaser clarified here in response to a question I had that the code for the new things is all available as part of the spec, and is under the GPL, here. So a freedoom-style resource wouldn't need to worry about any of the dehacked/decohack/etc., and just needs to replace the graphics and sounds. (Which is good because clean-room rewriting it to be demo compatible would be a massive pain in the ass.)

 

edit: I'm unsure about SBARDEF, it looks like it's needed for ID24 compatibility and it may need to be written from scratch, but that's also doable as the specs are fully documented.

[LuciferSam86]

Oooh makes sense. @plums .

And since id24 is under a GPL license even the next iterations like the future 1.0 will have a similar treatment. 

 

 

[Scorcher]

4 hours ago, GooberMan said:

Customisable skies, including PSX Doom/Doom 64’s fire skies 

I think it's important to elaborate more on this. Animated sky textures are already possible in modern sourceports (see Muumi's work) but the ID24 implementation seems to allow sky textures to be overlayed on top of eachother, which can create really cool stuff like that scrolling sky with the still-static mountains seen 5 minutes into the video.

[Devalaous]

3 minutes ago, Scorcher said:

I think it's important to elaborate more on this. Animated sky textures are already possible in modern sourceports (see Muumi's work) but this implementation seems to allow sky textures to be overlayed on top of eachother, which can create really cool stuff like that scrolling sky with the still-static mountains seen 5 minutes into the video.

 

Someone needs to animate stock Doom/Doom2/Final Doom skies using this. Give me wailing stretching souls in Doom 2's hell over those mountains. Twinkling stars on TNT's E2 sky. A burning city with billowing smoke for Doom 2's city levels!

[Gez]

14 minutes ago, Scorcher said:

I think it's important to elaborate more on this. Animated sky textures are already possible in modern sourceports (see Muumi's work) but the ID24 implementation seems to allow sky textures to be overlayed on top of eachother, which can create really cool stuff like that scrolling sky with the still-static mountains seen 5 minutes into the video.

Parallax skies have been a thing since Hexen.

 

And animated skies like Muumi's require a ton of frames. The classic firesky is automatically generated from just 32 color indices. Now the effect can be replicated with a ton of frames (there's a pack of PSX-style fireskies on Realm667), but it's more elegant when they're generated on the fly instead of having a hundred textures per sky.

[Doomy__Doom]

I don't see the point of JSON validation being required to hard fail on "metadata" field absence and/or contents. Community-wise, it's going to produce nothing but extra traffic to Editing Questions subforum with a typical answer being "copy sample from <wiki or smth link> and edit just the 'data' section".

 

It's fine for commercial port/wad to fail that way if it wants to, it's made by professionals who are good at (and have a job of) reading/following specs to a T. It does not help anyone in a wide public standard imo. People routinely fail at easier to make lumps.

 

 

Otherwise, lots of cool stuff!

[LuciferSam86]

38 minutes ago, Doomy__Doom said:

I don't see the point of JSON validation being required to hard fail on "metadata" field absence and/or contents. Community-wise, it's going to produce nothing but extra traffic to Editing Questions subforum with a typical answer being "copy sample from <wiki or smth link> and edit just the 'data' section".

 

It's fine for commercial port/wad to fail that way if it wants to, it's made by professionals who are good at (and have a job of) reading/following specs to a T. It does not help anyone in a wide public standard imo. People routinely fail at easier to make lumps.

 

 

Otherwise, lots of cool stuff!

I see your points. I mean those are just specs, I suppose with the next releases or even with the implementations can be implemented a validation of such data. 

[Redneckerz]

ID24HACKED is now on Wiki. That was a mammoth to convert, haha.

 

Existing ID24 page now also has Goober's features video.

[boris]

So the ID24HACKED format specifications states about indices:

 

Quote

The community range is not a free-use-by-anyone range. It is designed to be a range that the entire community can agree upon and implement in every source port. Any usage of this range that is not formally agreed upon by the community is considered in violation of ID24 [...]

 

So how's that supposed to work? If a single member of the entire community is not OK with using a index for something then it can't be used? A (not-yet-existing?) council decides what to do?

[Shepardus]

6 minutes ago, Ravendesk said:

I'm very confused about this, can you please clarify how this works?

So, features were developed in R&R which is GPL, then they got merged to the official port (KEX), then were modified in KEX and then the updated features will be merged back to R&R.

Doesn't this mean that KEX contains GPL code and should be GPL as well? What's the point in merging things back to R&R then and not just releasing KEX source?

I'm probably misunderstanding something though :)

I assume id licensed the code from GooberMan under a different license, which they can do since GooberMan wrote the code. The code's presence in a GPL codebase doesn't prevent its author from relicensing it under different terms.

[Ravendesk]

20 minutes ago, Shepardus said:

I assume id licensed the code from GooberMan under a different license, which they can do since GooberMan wrote the code. The code's presence in a GPL codebase doesn't prevent its author from relicensing it under different terms.

So since R&R is a fork of choco, and the code licensed was just from GooberMan it means that KEX is not a fork of R&R, but was implemented from scratch by GooberMan is that right? And the new features were implemented both for R&R and KEX only by GooberMan so R&R is not really relevant here at all, they just both happen to be maintained by the same person? Or what does "clean room" implementation means in this case?

[Redneckerz]

Wiki update:

• DEMOLOOP lump has a page now, based off the 0.99.1 spec
• ID24 page now includes the Mapping Additions document (Minus Line special table, i am just too tired haha). Mapping Additions highlights:
  • Interactive scrolling textures
  • Per-sector COLORMAPs without having to use Boom's specials
  • Floor and texture ceiling transformations

GAMECONF spec still needs to be brought over but that's almost as big as ID24HACKED, so i am leaving that for a bit.

 

[GooberMan]

Many of you have answered the questions and concerns correctly, so I won't repeat what has already been said there. Y'all are real heroes.

 

But yes, my implementation of Boom, MBF, MBF21, and even ID24 is incomplete. Things like the comp_soul flag not working are effectively bugs, usually because I didn't get around to implementing them.

 

UMAPINFO shenanigans are a bit of a different matter. I do have a huge rant about it being a spec that is 100% dependent on the implementation. But that's unimportant for the bugs reported here. If there's a field in UMAPINFO that seems to be not working, it's because of a stupid bug I found yesterday involving case sensitivity. Basically, my code ends up updating the wrong map structure internally. Derp.

[Gothic]

So, can you translate any color from the palette now, not just green?