PRBoom+ - remove rerecorded segments? Remove paused tics?

[greysondn]

Dayo!

I have some recordings in PRBoom+. Fantastic! Played them at a quarter speed.

Oh no! I saved and then loaded the save when I botched up. Now when I play back the video, you can see the botch and the load happen all the way through the fix and continue! Is there a way to change the video to only have a continuous video of the final takes that were made this way?

Oh no! I paused the game to go get a drink! Now there's paused frames in my video! Is there any way to drop paused frames from the demo?

Google alone didn't help me in my search. I didn't use the onforum search. I do use BizHawk on a regular basis; some of the terminology from that TAS crowd I can understand. I do know that almost anything in bizhawk while recording, if you load a savestate, it just records over your previous take. likewise, it's quite easy to just pause bizhawk itself if you need to stop for a while, as opposed to stopping the game's execution (as apparently was the case with Doom).

[kb1]

Remove Pauses? Yes, with a program designed to do so.

Remove Loads? Probably not, but a very slight maybe. The real question is: Does saving, and then reloading a game set up all of Doom's data structures perfectly enough for the demo to stay in sync? I have wondered this for a long time. My guess is that savegames don't save quite enough information to exactly replicate the game state enough to survive removal of the save/load command combo. It would be interesting to study, and it's on my long long list of to-dos...

[Edward850]

2 hours ago, kb1 said:

Remove Pauses? Yes, with a program designed to do so.

Not quite. Because of how Revenants work, you can only remove pause frames at multiples of 4. It can't remove all of them without introducing timing issues. Removing them from multiplayer is even more complicated (if not actually impossible) due to the input sequence.

2 hours ago, kb1 said:

Remove Loads? Probably not, but a very slight maybe. The real question is: Does saving, and then reloading a game set up all of Doom's data structures perfectly enough for the demo to stay in sync? I have wondered this for a long time. My guess is that savegames don't save quite enough information to exactly replicate the game state enough to survive removal of the save/load command combo. It would be interesting to study, and it's on my long long list of to-dos...

We already know the answer to this. Ignoring vanilla because its save format limitations should be obvious, PRBoom+ does not perfectly capture the blockmap thing orders (among possible others), which means reloading the game introduces an inconsistency, granted a mostly invisible one until the butterfly effect kicks in. PRBoom+ however can support recording from a previous point using a save or just recording the demo using -recordfromto. When using a save, PRBoom+ records the load event in the demo as if you just saved and loaded the game to maintain the logic.

[greysondn]

Both messy. I'm hopeful we'll reach a point when Doom is on par with many TAS emulators. I'd say I'm willing to implement it, but I looked at the code source release years ago and... well, frankly, it's beyond my depth. Maybe, maybe not right now, but I don't feel like doing it.

Is there any clean way to splice multiple recordings, so long as you make sure to observe multiples of four for revenants?

Lest I forget, thank you both for your responses thus far, and a blow of the raspberry to Doomworld because I've been here for docs and various things, I just prefer to lurk more. (Status was "registered just to make one post".)

[kb1]

20 hours ago, Edward850 said:

Not quite. Because of how Revenants work, you can only remove pause frames at multiples of 4. It can't remove all of them without introducing timing issues. Removing them from multiplayer is even more complicated (if not actually impossible) due to the input sequence.

We already know the answer to this. Ignoring vanilla because its save format limitations should be obvious, PRBoom+ does not perfectly capture the blockmap thing orders (among possible others), which means reloading the game introduces an inconsistency, granted a mostly invisible one until the butterfly effect kicks in. PRBoom+ however can support recording from a previous point using a save or just recording the demo using -recordfromto. When using a save, PRBoom+ records the load event in the demo as if you just saved and loaded the game to maintain the logic.

Wrong, and wrong:

1. PrBoom+ compensates for pause and menu tics, making it trivial to support removal of pauses, regardless of demo source or playback target. If target is vanilla, you can remove all but the remainder of # of pause tics mod 4, which is a maximum of 3/35 second, which is almost always a huge improvement.

 

2. Any of the possible issues are solvable. You say 'obvious': Obvious, in that you've found one thing that *might* cause sync problems, or obvious in that you've solved the issue, because the latter is the only interesting answer, as it introduces the possibility of doing some things that have previously not been feasible. What's obvious is that, if multiple ports can be made to play in-sync vanilla demos, sync-preserving save games are also possible, though they may not be there yet.

[Edward850]

Everything I talked about was in the now. You could fix the issues with PRBoom+, but at the moment it's imperfect. And you can't go back and change Vanilla saving without getting a time machine, going back to 1993 and breaking into id software and fixing it for them.

Also as you claim to be a source port author, you should be already quite well versed in Vanilla Doom's more glaring savegame bugs, i.e the fact it forgets all sectors you're standing in and can put you in inescapable holes, and that fact that all monsters lose their targets. These are even quite well documented phenomena. Hence obvious.

Edited August 11, 2018 by Edward850

[Linguica]

21 hours ago, greysondn said:

a blow of the raspberry to Doomworld because I've been here for docs and various things, I just prefer to lurk more. (Status was "registered just to make one post".)

 

You know, this has happened several times over the years where new users have taken the default title for a user with 1 post, i.e. "registered just to make one post", personally, thinking that it was done on purpose against them. I'm going to change it now while I am thinking about it anew.

[Pegleg]

14 minutes ago, Linguica said:

 

You know, this has happened several times over the years where new users have taken the default title for a user with 1 post, i.e. "registered just to make one post", personally, thinking that it was done on purpose against them. I'm going to change it now while I am thinking about it anew.

 

Actually, I didn't take offense at it; I got a chuckle out of the title. Perhaps that was because I did register for the express purpose of making one post, in that case, it was to ask a question that I knew the answer to. I'd been lurking for the better part of a year, almost entirely on the editing forum, before I saw the question and had to respond. 

 

Do users also get upset about being called Evil at 666 or Dank at 420? Do they get happy at being called Nice at 69? Oh well. Obviously, you can change it if you want, but I never saw the big deal.

 

By the way, @greysondn, welcome to Doomworld! Now that you have a presence, don't be a stranger.

[Linguica]

I removed "registered just to make one post" and also removed "Forum Newbie" since that sort of early-internet hazing environment is long, long out of date. Now you're just a New Member.

[Nine Inch Heels]

On 8/9/2018 at 7:30 AM, greysondn said:

Oh no! I paused the game to go get a drink! Now there's paused frames in my video! Is there any way to drop paused frames from the demo?

Best thing to do is to avoid pause-frames by hitting "ESC" rather than "PAUSE", because time spent in the menu during a recording session does not show in the demo at all.

[greysondn]

Quote

Linguica

 

Good on you. I don't think "Newbie" qualifies as hazing but "okay". (I'm from the newbs-are-not-n00bs school of thought.)
 

Quote

Pegleg

 

??? But I have presence in the TAS community and am working on glitch repros [youtube warning] in my free time? I was under the impression that, while the larger TAS community is distinct with a lot of overlap with DOOM speedrunning, the two communities are on good terms. In which case, I have been present - just not on these forums.

 

Quote

All

You seem to be struggling on consensus here. I... didn't realize something that seemed this direct would be that much of a question even to people I would assume play and know the engine much deeper than me. (I'm so hard for Doom 2 I cry in Doom 1 because "but muh super shotty!")

The nice thing about the 1.9 format is that it's easy to read (My current target for those demos to read is PRBOOM_6. compat mode 17. Compat 4 was low hanging fruit but I didn't expect it to be almost enitrely different for header format, among other things). Prboom+'s format is all but undocumented, buried in a forum thread miles deep as far as I could find. Still chipping away at that, but in no hurry as it's looking like...

• PRBoom+ wouldn't play nice with splicing? Was this even answered?
   
• I could remove the frames between a save command and a load command, but you seem to be saying that would desync because, among other things, the number of tics since the engine started is important (as opposed to since the run started; ie - the load action doesn't "roll back the tic counter")
   
• I could remove pause tics but, again, you seem to  be suggesting that would desync. Also, bad news, it does seem to pause on escape menu; next time I sit down to work on this instead of work for the local university I'll do some tests but I seem to recall noticing it was pausing while I was in the menu (during the recording) during playback. (Mouthful.)

How hard would it be to make a port that's stable enough that it could drop the tics without any assistance? I don't grasp why one can't basically lie - even to 1.9 - and give it an entirely clean take, even if it's rerecorded (that is, I don't understand why loading doesn't actually rewind time in the engine itself).

What I'm wanting would look like:

• Do nice amateur run of E1M1, 400 tics clean, save at 400
  • Engine tic counter: 400
• Play for 100 more tics, make a mistake
  • Engine tic counter: 500
• Load save
  • Engine tic counter: 400
• Finish after another 1000 frames
  • Engine tic counter: 1000
• Demo playback is fine, all tics consecutive from engine start, no saves, no loads apparent.
  • Engine tic counter during playback: 1000

May as well state you seem to be saying it would - at the present state of affairs - look more like:

• Do nice amateur run of E1M1, 400 tics clean, save at 400
  • Engine tic counter: 400
• Play for 100 more tics, make a mistake
  • Engine tic counter: 500
• Load save
  • Engine tic counter: 500
• Finish after another 1000 frames
  • Engine tic counter: 1100
• Demo playback is fine, all tics consecutive from engine start, with one save and one load in it.
  • Engine tic counter during playback: 1100

That last point is the one that makes me go "b-b-b-b-but..." - as in b-b-b-but why isn't state better encapsulated in saves and why can't we just pretend those 100 frames never happened anyway? (And again, this is based on what I understand the conversation, ultimately, to be saying here.)

And that doesn't even touch on pausing... :(

Edited August 11, 2018 by greysondn : Missed pegleg

[kb1]

@greysondn

You should be able to trim away pauses as long as you exclusively trim chunks of 4 consecutive pause tics. This keeps vanilla Revenants happy, as in vanilla they used some (gametic & 3) nonsense in the rev's missile code that will cause the demo to crap out.

 

For Load/Save, the prevailing theory is that Save doesn't capture the complete game state to the level of being able to preserve demo sync. So, doing a Save immediately followed by a Load leaves you in a slightly different state. Trimming those is probably problematic, though I've not personally verified it.

 

However, doing Load followed by the same Load? I would think that you'd probably be able to trim out everything between the first and second load without any issues (but someone might want to see each attempt.)

 

The PRBoom savegame system is much better than vanilla's - for example, vanilla's monsters would "go to sleep" because monster targets were not saved. So thing have gotten better. An important project is to find out exactly what is not being captured/restored, and improve the save, so it truly is a recreation of the game as it was. I imagine it's probably not that difficult. On the to-do list...

 

@Edward850When were we ever discussing saving in vanilla?

[greysondn]

What would a good test barrage look like for that? I could do the checks, albeit it'd be manual for now. It's not like it's THAT hard to make a small room with a feature or two in Doom Builder if we REALLY need to rig the situations.

 

---
Edit

Related note, how large could the state possibly be for Doom? Why not just save the entire game state instead of picking and choosing? "If it's a variable, we're gonna save it!"

A (vanilla) stage will run on a nintendo DS without problems, which means all relevant variables, plus engine, plus level data must fit into 4MB. It may be a lot of code to write... but I struggle to understand that 4 MB can't be found on a hard drive or in RAM nowadays O.o

Edited August 11, 2018 by greysondn : Well, state can't be THAT large, can it?

[vita]

Hello, always nice to see people interested in Doom TASing.

 

Firstly, you can use -recordfromto demo1 demo2 in prb+ and then press the join key (Q by default). No save tics, no pause tics at all (you won't need to press them, will you?).

 

Secondly, rerecording and slow-motion are not the best tools for Doom TASing. The Doom movies submitted to TASVideos by Akse and ClumsyDoomer all use frame-by-frame demo editing with XDRE (xepop's 2.14 src, my own half-assed repo and the binaries) and DRE. They're not easy to use, but you can always ask if you have trouble.

[greysondn]

It is a mistake to assume that merely because I wish to have a clean recording I am interested in TAS tools. Likewise, `-recordfromto` is less solution and more workaround. The questions asked were how to remove those frames and should be seen of value. That I can do better for tools to work with by playing a console port (For example, the PSX one [Hard edit here: or the homebrew DSDoom in emulator, which is built from PRBoom's source tree somewhere along the line]) should be a screaming red light that things don't have to be how they are presently and could - arguably - be better.

Solutions I'm considering workarounds and not true solutions:

• recordfromto
  • Can't be used to splice two independent takes together, such as a run of E1M1 into a separate run of E1M2.
     
  • Requires to watch basically the entire video, or to pray engine can skip forwards for you (I think prboom+ does this, but it's not really relevant)
    • In long playthroughs, the attrition rate for time lost to small mistakes rapidly piles up
       
  • Can't remove pause frames or save/load trash once it's already there
     
  • No true rewind, so have to be sure to catch the frame when it first passes by or you're stuck playing demo all over again - what madman would say that's a good solution?!?
     
• Segmented runs
  • But what if I want to start the stage with the shotgun and rocket launcher I picked up a little early from the secret? Let alone it should be quite reasonable to play back all of Doom as one recording anyway.
     
  • And it's absurd to say segmented runs reflect the actual game purely, because players would have stats carried over between stages, after all.
     
• Current save/load in Prboom
  • Shows the frames up to the load, then executes the load as an input in the tic. Non-ideal for watcher and runner both
• Current frame-by-frame run creation tools
  • So my choices are a hammer made for something smol that my hand is too large for... and a sledgehammer that I can't really lift.
     
  • To be clearer, just because save/load right now is a bad state doesn't mean that purely playing frame-by-frame is the solution. There's a lot of space between the two that is just being left completely blank here.
• Use case
  • I am a decent player, but I don't want to repeat stages over and over, so I load games tool-assisted to make them easy. If we agree that I could get to a certain point that I've saved at if I restarted the game, then we can agree it's a waste of time to make me redo the entire game. In other words, saves are used as checkpoints more than anything. I keep trying portions of stages over and over. I get a decent run and want to post it on youtube, but I don't want to waste viewer's time and I don't want to waste time perfecting the run right this instant (and if we're literally talking about me, ever). It'd be great if the portions that would be in a final take in an emulator like Bizhawk were all that were in the final video.

I'm certainly open to questions but I don't grasp what is hard here. Prboom would need better save load to even start on this, it seems like it wouldn't take much ram to save the entire state if nothing else, but even if that's not an option I'm open to creating a test barrage and tools to work with the recordings, and to manually work with the tests in our present state (to all appearances, incapable of being analyzed automatically in the engine itself). Let's talk and make this happen! ... or at the very least, talk about actual solutions, instead of workarounds or solutions looking for problems to be applied to.

[Linguica]

The thing is that since Doom has always had built-in demo recording, and these demos are in a very bespoke format, the tools for adding anything new to demo recording functionality have always been sparse at best or nonexistent at... normal.

 

Save / load are not demo safe and should never be counted on to be demo safe. If you want demo-safe rewinds, you need to save the *entire* game state, which is a subtly different set of data from the game state as stored by the savegame routines. This is at least theoretically not SO hard to do - the Doom engine stores (almost) all game state in a list of "thinkers" which includes the current state of all things in the map, all changing / flickering lights, all doors and other moving sectors, etc. If you store this for every frame (or, more likely, for every 4th frame, let's say), the size of that gamestate at any moment will be large enough to matter, but not terribly large - basically the size of a savegame file, which is usually in the tens of kilobytes. With modern computers it shouldn't be that taxing to just hold them all in memory as you play, and at worst you might have to design a sort of I-frame and B-frame system to compress the data across tics.

 

XDRE clearly has figured out how to fully serialize game state, but it's obviously not real time. At least one person has tried putting demo rewind into a source port before, but it's never been done in a complete and useful way to the best of my knowledge - again, because Doom already has demo recording and rewinding is a feature that is not trivial to add and is of unknown benefit to Doomers.

[kb1]

@greysondn and @Linguica:

I'm with you - I am very interested in your ideas and requirements, and, yes, I do believe that what you want to accomplish is not only possible, but that we're 90% there now.

 

We are currently saving just about everything that humans can detect. What may be missing is some of the ugly global intercept state, and the order of various lists. You can't just save whole memory blocks, without converting pointers to indexes on save, and back to pointers on load. A very crude but interesting test would be to implement a per-frame Save/Load combo (that's not recorded in the demo) as a toggle-able option, and compare a demo with the switch toggled off and on. This could be used to prove that we're missing some state. Better yet is to add a third savegame function: "Verify", that can test a savegame file against what's in memory.

 

I have committed to finishing my real-life project by the end of the month, and, God help me, if I can get it done, I'll finally be able to devote some serious energy to Doom. I've been thinking about rewindable demos for a while now, as well as perfect state Save/Load, automatic demo recording, and lots of other goodies. What I have accomplished is my soon-to-be-released SyncDebug system that assists port developers in being able to accomplish vanilla demo sync, by rigorously comparing frame-by-frame, thing-by-thing state to a known working port. By it's very nature, it presents a hyper-detailed view of all things state-related, in an easily-readable form. I believe this tool will greatly simplify the task of discovering what's missing in the savefile, and it should be useful for some of the other tasks outlined above.

 

If we could accomplish a demo-preserving save, pseudo-rewindable demos become trivial:

1. Auto-save every X seconds or so.
2. Rewind to the desired X-second chunk, then speed thru the max (X*35) frames, up to the desired frame.

You couldn't actually play the demo backwards...that's better accomplished with video tools.

 

You determine 'X' by how fast your engine can blaze thru gametics without rendering, vs. how comfortable you are with waiting for the rewind to resolve. You could get really fancy and generate these save states on-the-fly as the demo is watched. Or, taken to the extreme, the save states could be stored in memory, and you could interlace a standard speed playback with internal high-speed save generation. This would cache forward frames internally as quickly as possible, providing near-instant fast forward minutes ahead, followed by near-immediate rewind to any point. After a short cache period, you could enjoy random access to the entire demo (within some limits).

 

These save states could be generated on-the-fly, and/or be stored within the demo file. It's not the most efficient thing to do, but it sure beats trying to make Doom reversible...

 

Hopefully, you can tell that I'm pretty passionate about the idea. I just hope I can release some pressure from my schedule. I've been planning these features and more for a long time now... We'll be there one day. It's nice to know that other people are thinking about the same kinds of stuff :)

 

It would be so cool to be able to record a demo like that - it's like how recording studios put together many songs...by splicing in a piece of this guitar solo take with that one, trimming out a sour note here and there. Maybe for Doom demos, some rules need to be considered. I guess such demos would be categorized as TAS demos, but maybe a different class of TAS?

 

[greysondn]

I see no reason not to consider them the same as any other TAS demo. But, regardless...

... When you get around to looking at this and hit your tool's public release, let me know. I looked over PRBoom's code and while I get the gist of it I'd have to basically translate it to a form I understand better to really get a good feel for the internals. Regardless, I'm still available for testing and external tooling.

[kb1]

On 8/14/2018 at 6:33 PM, greysondn said:

I see no reason not to consider them the same as any other TAS demo. But, regardless...

... When you get around to looking at this and hit your tool's public release, let me know. I looked over PRBoom's code and while I get the gist of it I'd have to basically translate it to a form I understand better to really get a good feel for the internals. Regardless, I'm still available for testing and external tooling.

I don't know - it seemed like it'd be a tiny bit less "TAS" then TAS, but maybe not.

 

Sure, I'll let you know. Tool's ready, gotta finish docs. Thanks for the offer to do testing, though that's gonna be a bit further down the road.