Restoring vanilla save targets

[Randy87]

I noticed that Doom blasts entire structures into the savegame file.
This includes any pointers that the structure may have contained.
These pointer fields are null'd or replaced on loading a savegame.
This leads to mobjs losing their targets.

However, the old pointers do provide useful information.
All thinkers are serialized into the savegame file.
Each thinker has a pointer to the next and previous thinker.
Using this data, we can generate a table of old links.
This information can be used to restore target data.

   cur_addr old_addr old_next old_prev old_targ
 0 0304AF14 03039008 03039154 005614C4 00000000 Mobj
 1 0304AFC8 03039154 03039208 03039008 00000000 Mobj
 2 0304B07C 03039208 030392BC 03039154 00000000 Mobj
 3 0304B130 030392BC 03039370 03039208 00000000 Mobj
 4 0304B1E4 03039370 03039424 030392BC 00000000 Mobj
 5 0304B298 03039424 030394D8 03039370 00000000 Mobj
 6 0304B34C 030394D8 0303958C 03039424 00000000 Mobj
 7 0304B400 03042E10 03039640 030394D8 03039008 Mobj
 8 0304B4B4 0303958C 005614C4 03042E10 00000000 Mobj
 9 0304B568 03039640 03042DD0 0303958C 00000000 Glowing Light
10 0304B59C 03042DD0 03042E10 03039640 00000000 Door
11 0304B5DC 03042E10 030478F4 03042DD0 00000000 Ceiling

We can match index 7's old target to index 0's old address.

Here's a quick hack source tree of Chocolate-Doom 2.2.1 that I made to test the theory.
Does not include binaries.
Chocolate-Doom-2.2.1_TargetRestore.7z

[Maes]

I have already implemented this feature in Mocha Doom since 2011, and the theory was known even before that, by fraggle if I recall.

[kb1]

Randy87 said:

...
Using this data, we can generate a table of old links.
This information can be used to restore target data...

Maes said:

I have already implemented this feature in Mocha Doom since 2011, and the theory was known even before that, by fraggle if I recall.

...but, yeah, that's a cool (re)discovery :) If you think about it, id got really lazy on this one. Not only does the game forget targets, but it wastes precious savegame memory. But, if the memory has not been wasted, your restore magic would not work.

[Linguica]

paging xttl to hack this into the doom2.exe binary...

[fraggle]

Note that I believe this won't currently work on Chocolate Doom savegames because pointers are all written as null pointers. I should fix that.

[Quasar]

fraggle said:

Note that I believe this won't currently work on Chocolate Doom savegames because pointers are all written as null pointers. I should fix that.

Question would be how to handle x64, since the pointer fields have to be 4 bytes for vanilla compat, but 64-bit pointers could potentially alias each other with the top 4 bytes stripped off - especially on secure heap OSes like BSD where allocations can be spread out through the address space. (ie., 0x000000cd4fdb23c0 vs 0x000000fe4fdb23c0)

I'd probably suggest that an increasing unique ordinal be written, since vanilla doesn't care what's in that field and will overwrite it anyway. It would require a hash table of pointer values to indices during serialization, however.

[fraggle]

Quasar said:

Question would be how to handle x64, since the pointer fields have to be 4 bytes for vanilla compat, but 64-bit pointers could potentially alias each other with the top 4 bytes stripped off - especially on secure heap OSes like BSD where allocations can be spread out through the address space. (ie., 0x000000cd4fdb23c0 vs 0x000000fe4fdb23c0)

I'd probably suggest that an increasing unique ordinal be written, since vanilla doesn't care what's in that field and will overwrite it anyway. It would require a hash table of pointer values to indices during serialization, however.

My thoughts exactly. I think the 64-bit case is why I started writing NULL pointers (at the time I wasn't expecting them to ever be used for anything). The solution you describe is how this would need to be implemented.

[fabian]

It might be interesting to know at this point that Crispy's ability to restore target pointers when loading savegames works quite similarly: When saving a game, each mobj-thinker in the thinker chain gets a unique integer assigned in increasing order. This index is saved in the savegame instead of the actual thinker pointer. When reloading the savegame, after all thinkers have been restored, the indices are translated back to the target pointers.

The code is around here:

https://github.com/fabiangreffrath/crispy-doom/blob/master/src/doom/p_saveg.c#L1728

[Maes]

Quasar said:

I'd probably suggest that an increasing unique ordinal be written, since vanilla doesn't care what's in that field and will overwrite it anyway. It would require a hash table of pointer values to indices during serialization, however.

In fact, in Mocha Doom I used the default Object class hash as a "pointer" value when saving -unique enough to work, but an ordinal would be a better solution and easier to decode.