UDSF - Universal Doom Save Format

[jval]

As mentioned here by fraggle, is there any interest in creating a cross-port save game format specification?

[RestlessRodent]

Many modern source ports are very different from each other that a common save format would be extremely complicated. Although there are many similarities between what is in the game state of source port there are many differences. Going from one source port (ReMooD for example) to another (ZDoom for example) would have a loss of information along with if you save the game in the other port and come back it would possibly lose more information. You can think of saving JPEGs or reencoding OGG over and over again.

It would be much work, because you would have to define fallback states for things.

[Graf Zahl]

Sorry, won't happen.
A large portion of the savegame contents are implementation specific and not easily transferrable.

Just as an example, ZDoom uses Hexen-type specials for linedef actions internally, it's plain and simply impossible to store them in a cross-port-compatible way so that they can be used by ports that only handle the original Doom specials.

[Gez]

At most it could be possible for ZDoom to read such a save and convert it to internal format on the fly, but definitely not write it. The same would be true of any other advanced port. If you can define new actor types, then how could you ensure that this new actor type would work in a different port? Content definition languages would have to be universal first before a save format can be universal.

And actors are only one part of the issue. There's also the map geometry. Sector effects, thinkers, linedef types, etc. That, too, would need to be universal first.

And if these things become universal, then what do you do when you want to add a new feature? Roll it on your own and break the standard? Suggest it to the other port maintainers so that it can be adopted simultaneously by everyone? What if one of them doesn't like the feature and doesn't want it to become standard?

So because of that, having a universal save format that can handle advanced new features is pretty much impossible and undesirable. The only way to make something universal, then, is to restrict it to a common minimum subset of features that every port ought to support; and then once you do that you discover this format already exists, in a way: it's the vanilla save format.

[VGA]

Someone could write a tool that converts saves between the formats.

[RestlessRodent]

VGA said:

Someone could write a tool that converts saves between the formats.

It could be done, however you still have port specific stuff and you have architecture dependent stuff. So for an example with PrBoom, you will have to add support for 32-bit, 64-bit across various architectures such as PowerPC, ARM, MIPS, SPARC, x86, etc. with adjustable structure alignments along with byte order.

The quickest and easiest route would be to rewrite the savegame code of a source port so that it is deterministic regardless of the architecture.

[kb1]

jval said:

As mentioned here by fraggle, is there any interest in creating a cross-port save game format specification?

Do you mean save from PortX, load in PortY, and play the game? Or do you mean save in PortX, then load the save game in a universal save game viewer app? It's not a real practical idea, but, if it were me, I'd start with this back-of-the-envelope idea:

Save 2 files: DOOMSAV0.DSG (original vanilla-like format), and MyPortName0.PSG containing wrappers for port specific stuff. It'd almost have to be like a WAD file, or a .zip, or a PNG, where you have place markers for the different types of data.

OR... the full-blown all text UDMF approach. I like the idea of including the vanilla format though. Might give ports a chance to load something. Seems very messy to me. I'm all for universal formats, but I've never had any luck convincing the majority, even for simple stuff. UDMF should be considered a HUGE accomplishment, both for being technically sound, but also for getting some community support!

[VGA]

The popular source ports use architecture-dependant save formats? What the heck do they do, dump all the structures from memory to disk?

[Linguica]

VGA said:

The popular source ports use architecture-dependant save formats? What the heck do they do, dump all the structures from memory to disk?

Prrrrretty much

[Maes]

The problem is that if you have a "normal" source port written in a "normal" language like C/C++/Delphi/Assembly etc., the natural course of action when loading/saving pretty much anything is simply to read/write raw structures from/to memory, especially with what concerns strictly the in-memory structures of YOUR OWN port (e.g. savegames).

There's no reason or incentive to go through a field-by-field marshaller/unmarshaller in most cases, not even for reading "legacy" vanilla format, little-endian stuff like existing PWAD files (at least not on Intel or ARM). I had to do just that in Mocha Doom though, so for me it would -relatively- trivial to add a conversion layer on top of the -already existing- mechanism. But for everybody else it would be a needless complication.

[chungy]

Big endian systems are currently very uncommon. Even architectures like ARM, MIPS, and POWER where the CPU can operate in either mode, they are most frequently run with little endian kernels and programs.

Most ports (even ZDoom I'm sure) go through the effort to keep WAD loading working on big endian systems, but by large... save games aren't usually transported across systems, or even across ports. It's really kind of a minute issue. Chocolate Doom goes through the effort of keeping save games vanilla-compatible, but it's basically alone in that, and it's an artifact of the project's strict goals.

[Maes]

Actually, simply by porting the vanilla linuxdoom v1.10 code and not changing a line in the savegame code, it automatically becomes "savegame compatible" with vanilla - at least barring endianness and word alignment issues between different compilers. But on a little endian platform with a 32-bit word alignment, it should be compatible.

So rather than a deliberate goal, it's actually the 'natural state' for any straight port of the SC. Once you start cobbling your own stuff on top though...that's another story.

[jeff-d]

chungy said:

Big endian systems are currently very uncommon.

They might be uncommon, but I have one!

Maes said:

There's no reason or incentive to go through a field-by-field marshaller/unmarshaller in most cases

Once you have a marshaller/unmarshaller it's relatively easy to always save in little endian format.

I still have some very old save games written by an old big endian system. As luck would have it the first record to be loaded/saved is the player. There are some variables that are saved as 32 bit (e.g. health) that can easily be determined to be the 'wrong' way round, hence the marshaller/unmarshaller can swop the fields on loading.

However, I would suggest that a UDSF file should be in plain text, not unlike ID solved the problem in Quake. All ports would be able to read/write the Vanilla (and possibly BOOM) compatible stuff. Other port extensions would have to be ignored when loaded by a different port.

[Graf Zahl]

jeff-d said:

All ports would be able to read/write the Vanilla (and possibly BOOM) compatible stuff.

No, they can't. As I said before, ZDoom does not internally use linedef actions in a format that's compatible with Doom's original format so that alone would pretty much prevent it from saving such files.

And I'm quite certain there's other gotchas as well if you start to look for them.

[chungy]

Maes said:

So rather than a deliberate goal, it's actually the 'natural state' for any straight port of the SC. Once you start cobbling your own stuff on top though...that's another story.

Take another look at Chocolate Doom. It rather deliberately goes out of its way so that 64-bit and big-endian builds can save and load games in a manner 100% compatible with vanilla Doom. As you said yourself, this isn't a property that Linux Doom itself has.

[Quasar]

Eternity's saves are endian agnostic, but they are not currently portable between 32- and 64-bit architectures.

[RestlessRodent]

Doom Legacy and C ReMooD have deterministic saves, since otherwise you would be unable to play multiplayer games with other architectures and bittage. Since when you join a network game the server sends you the save game which is then loaded by the client.

In Java ReMooD it is going to be even easier as I can use lambdas, CallSites, and interfaces to handle read/write of data only using a single set of code for the most part. Also, Java usually defaults to big endian data storage (also known as network byte order), so if I use DataOutputStream this would be the case.

[Maes]

chungy said:

Take another look at Chocolate Doom. It rather deliberately goes out of its way so that 64-bit and big-endian builds can save and load games in a manner 100% compatible with vanilla Doom. As you said yourself, this isn't a property that Linux Doom itself has.

Well, I said "at least barring endianness and word alignment issues". But even assuming a completely unaltered source code, with no preprocessor directives, no platform specific blocks or macros etc., then at least if compiled as-is on a 32-bit little endian architecture with a 32-bit word alignment, it should result in identical savegames with vanilla Doom.

[Linguica]

I think any time that anyone talks about cross-compatibility stuff in Doom source ports you can pretty much mentally append "not including any ZDoom-based ports obviously"

[wesleyjohnson]

** With as much trouble as I have keeping savegame format compatibility with additions to DoomLegacy versions, I do not see how a Universal Format could keep up with so many ports. DoomLegacy intends to keep making additions to its capability, which will affect savegames.

DoomLegacy has an improved Savegame format.
- Readable file header in plain text,
with DoomLegacy version, samegame format version, game, wad file, command line switches, and a user description.
It can be read using any editor or MORE/LESS.
- If a savegame is selected with the wrong game or without a need wad file, DoomLegacy informs the user of the problem, and refuses to load the savegame.
- Section sync, with sync detection.
- Optional sections, so section additions do not break the save game format again.
It detects section byte after the Sync code, which indicates the next section.
Currently there is only one optional section, and it has to be in a particular
position in the file.
- Pointer conversion to an index of mobjs. A pointer conversion index table is stored early in the savegame. Pointers are not stored in the savegame anymore.
Changes in memory allocation or structuress does not affect savegame loading.
- The savegame loading can detect unsatisfied mobj pointers on load and convert them to NULL (happens when a monster is killed and there is still a reference around).
- all fields are converted to a byte stream by the savegame function.
The savegame function protects the savegame format from changes in the rest of the code, especially changes to structures.
- There are no marshallers, nor any binary images, nor any self storing structures.
- Intent to keep some backward compatibility with savegames of the previous versions of DoomLegacy. It may not restore the game perfectly, but do something reasonable
make it possible to keep playing.

[fraggle]

Maes said:

Well, I said "at least barring endianness and word alignment issues". But even assuming a completely unaltered source code, with no preprocessor directives, no platform specific blocks or macros etc., then at least if compiled as-is on a 32-bit little endian architecture with a 32-bit word alignment, it should result in identical savegames with vanilla Doom.

Actually I believe Watcom didn't align fields so if you align to 32-but boundaries it won't work. You need to provide the attribute to pack structure fields (Chocolate Doom already does this for most structs read from WADs.

You've covered most of the other variables but there are still others. For example enums have to be represented as 32-bit fields; Chocolate Doom had a bug recently because on ARM gcc can sometimes represent enums as smaller fields.

In general though the "dump structs to disk" approach just isn't workable - it's based on compiler and architecture specific behaviour and can't be relied upon to give consistent results. That's why for Chocolate Doom I replaced it with (autogenerated) functions that explicitly read and write each field.

[Maes]

Now that I rechecked it, the savegame code actually went through a painstaking marshalling of select game's structures, trying to write as little as possible to disk.

https://github.com/id-Software/DOOM/blob/master/linuxdoom-1.10/p_saveg.c

Not only that, but it also had an explicit padding function, which was called every once in a while.

// Pads save_p to a 4-byte boundary
//  so that the load/save works on SGI&Gecko.
#define PADSAVEP()	save_p += (4 - ((int) save_p & 3)) & 3

so that takes care of compiler-enforced struct alignment (or lack there of). Notice how they use a byte* pointer for advancing the save, so even if that means the compiler generates inefficient code for byte-wise memory access, it should still be portable.

Most writes were of whole fields (integers, fixed_t numbers, booleans, indices, enum values etc.), and very rarely was a whole structure dumped to disk (actually, to a memory buffer) as-is. I think only mobj_t and player_t are even dumped "as is" to disk, and even then, the PADSAVEP() macro keeps everything aligned to 32 bits. Only on a 64 bit architecture this could break, if the player_t or mobj_t structs are alignable to 32 bits but not 64. Or elements like the enum representation bug (or feature?) you mentioned, but even those can usually be fixed by configuring the compiler, the goal being to make it behave as closely to pure 32-bit as possible.

In retrospect, this approach allowed them to save some memory and disk space (since both were at a premium) but for a modern source port, not even that degree of marshalling made sense.

[Coraline]

Universal saves sounds like a serious waste of time due to all the damn variables and conditions one would need - it would be outrageous. Though I won't stop anyone.

Good luck trying to integrate 3DGE's savegames, that's all I'm going to say.

[fraggle]

Maes said:

Now that I rechecked it, the savegame code actually went through a painstaking marshalling of select game's structures, trying to write as little as possible to disk.
https://github.com/id-Software/DOOM/blob/master/linuxdoom-1.10/p_saveg.c

Not really, look more closely. The only function that's really true for is P_ArchiveWorld(). The rest of it is raw structures. You might be misled by some of the bits in P_ArchiveSpecials() that reset or munge certain fields.

[kb1]

Graf Zahl said:

No, they can't. As I said before, ZDoom does not internally use linedef actions in a format that's compatible with Doom's original format so that alone would pretty much prevent it from saving such files.

And I'm quite certain there's other gotchas as well if you start to look for them.

"Prevent" is a little bit harsh. We are discussing computers and programming, after all.

Linguica said:

I think any time that anyone talks about cross-compatibility stuff in Doom source ports you can pretty much mentally append "not including any ZDoom-based ports obviously"

You know, with a different philosophy, ZDoom could bring the masses together and become the go-to-port. If the ZDoom team was interested, we could revive the Doom Standards committee, and get a level of compatibility across ports never before seen. ZDoom is in a unique position of authority to be able to make that happen. I'd leave my port in a ditch on the side of the road, if ZDoom worked with others on standards (instead of being their own standard, so to speak). Not trying to start a debate - I just think it's really unfortunate (and it could be solved).

[Edward850]

[jeff-d]

Maes said:

Not only that, but it also had an explicit padding function, which was called every once in a while.

// Pads save_p to a 4-byte boundary
//  so that the load/save works on SGI&Gecko.
#define PADSAVEP()	save_p += (4 - ((int) save_p & 3)) & 3

I hated that code, far too clumsy!

This was my "improved" version:

// Pads save_p to a 4-byte boundary
//  so that the load/save works on SGI&Gecko.
#define PADSAVEP()	save_p = (byte*)(((int)save_p + 3) & ~3)

[RestlessRodent]

Might be better as:

#define UINTPTR_C(x) ((uintptr_t)(UINTMAX_C(x)))
#define PADSAVEP()	save_p = (byte*)(((uintptr_t)save_p + UINTPTR_C(3)) & (~UINTPTR_C(3)))

Since if you are on a 64-bit CPU and save_p is really high up, you mask away the high bits and end up writing in the wrong location.

[jeff-d]

GhostlyDeath said:

Since if you are on a 64-bit CPU and save_p is really high up, you mask away the high bits and end up writing in the wrong location.

Yes, I've already done that but omitted it for clarity.

[Gez]

kb1 said:

You know, with a different philosophy, ZDoom could bring the masses together and become the go-to-port. If the ZDoom team was interested, we could revive the Doom Standards committee, and get a level of compatibility across ports never before seen. ZDoom is in a unique position of authority to be able to make that happen. I'd leave my port in a ditch on the side of the road, if ZDoom worked with others on standards (instead of being their own standard, so to speak). Not trying to start a debate - I just think it's really unfortunate (and it could be solved).

Nah, not going to happen.

Take a look at the Universal Doom Map Format. It was a pretty cool idea, right? A common, standard text-based map format that ports can extend with new map editing features without creating conflicts or other compatibility issues. This was pioneered by Quasar and developed with Graf Zahl and CodeImp. Fast forward to now: it's basically a ZDoom-exclusive feature. In a few months the standard will be eight years old, and the only port not directly derived from ZDoom that has adopted it is Vavoom.

Furthermore, there's been very, very little enthusiasm from other port developers whenever there were talks of creating a "boom+" standard. Typically, Graf and Quasar would mention it, and then they'd be joined by a chorus of crickets.


A universal save format is even worse than a universal map format, because it maps directly to how the game data is represented internally. On a map, you can say "there's an imp in this position, with these spawn flags"; in a save you have to also say whether it has woken up or no, what is its target, what are its momentum on the three axes, and all the myriad other variables that define a mobj in memory, like reactionTime and whatever. It's no longer "an imp", it's "Bob the imp, with all his hopes and dreams" and when you want to save that you have to save everything that makes it work in your port. What happens when you have a port-exclusive feature that you've added to imps which tracks the amount of ice creams they offer if you befriend them? Either you add an icecream counter to the format, or you don't. If you do, other ports will not know what to do with that field (they can ignore it harmlessly, though) but if you don't, then you basically break your saves because this feature will stop working correctly when saving and reloading.

Now suppose you save Bob the imp, and you do write his icecream counter. The save goes to the Cloud, and you load it with another port which does not have icecream imp support. You continue playing, though Bob is no longer your friend, and then you save to the Cloud again. You go back to your first port, load your progress, Bob is not magically your friend again because the other port didn't keep track of Bob's ice cream counter; and your save is basically broken even if it works.

Conclusion: a universal Doom save format would not help ports with advanced features.

Corollary conclusion: since it would be useless for advanced features, it would only support the common minimum set of features.

Corollary to the corollary: this format already exists, it's the save format as defined by vanilla running on a PC DOS platform.