What's the difference between 1.0 and 1.2 demos?

[kb1]

Does anyone know if there is a fundamental difference in format between version 1.0 and 1.2 of Doom demos. Based on entryway's awesome 1.2 demo work, I can watch IWAD 1.2 demos, but IWAD 1.0 demos desync, with "GREEN IS TURBO!" messages.

Is it just me, or has anyone had better luck with those?

[LogicDeLuxe]

id extended the header in v1.2 to support multiplayer, multiple maps and multiple lifes. The details should be in the doom specs document, iIrc. There is also a converter somewhere, I don't remember exactly.

Converting between various versions would be hit and miss though, since demos weren't designed with upgrades in mind. A little tiny detail changed in the engine, and the demo can desync completely.

The best thing you can do for watching old demos is downgrading your Doom version. You would probably need a DOS emulator as well.

[kb1]

LogicDeLuxe said:

id extended the header in v1.2 to support multiplayer, multiple maps and multiple lifes. The details should be in the doom specs document, iIrc. There is also a converter somewhere, I don't remember exactly.

Makes sense that they changed it, but it's not obvious to me where the change occurred while examining the two side-by-side in hex.

The first 1.0 demo contains:
03 01 02 01

I assume that means skill 3, E1M2, with player 1 active.

But, the first 1.2 demo has exactly the same header.

Can I bother you (or anyone else) for the specs, and/or the converter? A quick search for the converter did not reveal anything useful.

I'm actually interested in supporting the playing of the original 1.0 demos in a source port.

LogicDeLuxe said:

Converting between various versions would be hit and miss though, since demos weren't designed with upgrades in mind. A little tiny detail changed in the engine, and the demo can desync completely.

Yeah, I imagine I'll encounter some desynchs, but, getting "GREEN IS TURBO" messages indicates to me that I'm not even reading the format correctly.
Perhaps the multiplayer additions are adding 3 bytes to the header, one for each possible additional player.

[Maes]

I don't remember exactly how the raw demo lumps are parsed, but the cheating -or not- state is encoded in the player info, and it may take but one bit out of place to trigger the cheat. I'm not sure if the turbo cheat even existed before 1.2

[Never_Again]

First, let's clear up a misunderstanding invoked by the title of this thread: there's no DOOM v1.0. The original shareware release was v0.99, the next release went to v1.1 at once, similar to the later v1.2 -> v1.4b jump.

Second, DOOM demos v0.99, v1.1 and v1.2 use the same 7-byte header. (The 13-byte header was introduced in the v1.4 beta - not counting the leaked v1.3 betas whose authenticity is rather dubious). The extra six bytes account for the following:

• demo version
  
• solo/coop/DM mode flag
  
• -respawn flag
  
• -fast flag
  
• -nomonsters flag
  
• viewpoint
  

Note: multiplayer data in the demo header was not introduced in v1.2, it was there from the start.

Anyway, not just the header, the format between the first three versions seems to be the same.

Examining the demo with LMPC reveals there are numerous tics with movement values well over 50 (maximum speed with speed on):

GF63 TL1                #    44 (1.25s)
GF113 TL6               #    45 (1.28s)
GF113 TL12              #    46 (1.31s)
WT                      #    47 (1.34s)
GF91 TL14               #    48 (1.37s)
GF18 TL7                #    49 (1.39s)
GB2 TL8                 #    50 (1.42s)
GB9 TL8                 #    51 (1.45s)
GB106 TL18              #    52 (1.48s)

Given that -turbo didn't exist in v1.2 and earlier, this could mean only one thing: the movement value range changed after v1.2.


Here is the entire demo section from Unofficial DOOM Specs v1.666:

[8-6]: DEMOs
============

  If you start DOOM and do nothing, after a few seconds, it automatically
shows a demo of play on some level. Also, external demos can be recorded
and played back by using the command line parameters explained in the
README and/or the DOOM FAQ. All external demos have a .LMP extension
which the DOOM OS attaches; you only type the [demoname] without the
.LMP extension.
  The DOOM.WAD lumps DEMO1, DEMO2, and DEMO3 are in exactly the same
format as these external .LMP files. Strictly speaking, the "demo"
format should not be called the "LMP" format, because any external
file without a wadfile header, i.e. it is just raw data, is a "lump"
and deserves the .LMP extension.

  A DOOM demo has three parts:

  (1) header - 7 or 13 bytes
  (2) data recording player moves - 4 bytes per player per gametic
  (3) quit byte - equals 128 (0x80)

(1) There are two different kinds of header depending on the version of
DOOM used to record the demo. Versions up to 1.2 use a 7-byte header:

  byte  range   purpose

0       0-4     skill level. 0="I'm too young to die", 4="Nightmare!"
1       1-3     episode.
2       1-9     mission/map.
3       0-1     player 1 is present if this is 1.
4       0-1     player 2.
5       0-1     player 3.
6       0-1     player 4.

  Versions after 1.2 use a 13-byte header:

byte    range   purpose

0       104-106 version. 104=1.4 beta, 105=1.5 beta, 106=1.6 beta or 1.666
1       0-4     skill level. 0="I'm too young to die", 4="Nightmare!"
2       1-3     episode. In DOOM 2 this is always 1.
3       1-32    mission/map/level. In DOOM 1, it's 1-9. In DOOM 2, it's 1-32.
4       0-2     mode. 0=single or cooperative, 1=deathmatch, 2=altdeath
5       0-      respawn. 0=no respawn parameter, (any other value)=respawn.
6       0-      fast. 0=no fast parameter, (any other value)=fast.
7       0-      nomonsters. 0=monsters exist, (any other value)=nomonsters.
8       0-3     viewpoint. 0=player 1's status bar, ..., 3=player 4.
9       0-1     player 1 is present if this is 1.
10 0x0a 0-1     player 2.
11 0x0b 0-1     player 3.
12 0x0c 0-1     player 4.

(2) The player-move data is recorded in 4-byte chunks. Every 1/35 of a
second is a gametic, and for every gametic, there is one 4-byte chunk
per player. So the time duration of a demo (in seconds) is approximately
equal to its length in bytes divided by (140 * number_of_players).

  The four bytes recording each player's actions are:

  (a) Forward/Backward Movement.
  (b) Strafe Right/Left Movement.
  (c) Turn Left/Right.
  (d) other actions - use/activate, fire, change weapons.

  The first three are signed bytes (i.e. of type <char>).

  (a) Ranges from -127 to 127, negative numbers are backward movement,
      positive numbers are forward movement. Without the -turbo option
      above 100, values outside -50..50 cannot be achieved. With a
      keyboard or joystick, these are the regular values:

      Move forward:   25 (0x19)   with Speed on:  50 (0x32)
      Move backward: -25 (0xE7)   with Speed on: -50 (0xCE)

      Fancy mouse use can achieve any number in the range.

  (b) Ranges from -127 to 127, negative numbers are left-strafe movement,
      positive numbers are right-strafe movement. The keyboard values are:

      Strafe right: 24  (0x18)    with Speed on:  50 (0x32)
      Strafe left: -24  (0xE8)    with Speed on: -50 (0xCE)

  (c) Ranges from -127 to 127, negative numbers are right turns, positive
      numbers are left turns. The keyboard values vary from version to
      version, but are all in the range -5..5, and that's with Speed on.

      Using the mouse can achieve much higher numbers here. I doubt if
      the maximums of 127 and -127 can actually be achieved in play,
      though.

  (d) the bits of this byte indicate what actions the player is engaged in:

      bit 0     Fire current weapon
      bit 1     Use (a switch, open a door, etc.)
      bit 2     Change weapon to the one indicated in bits 3-5:

      bits 5-3 = 000 Fist or Chainsaw
		 001 Pistol
		 010 Shotgun
		 011 Chaingun
		 100 Rocket Launcher
		 101 Plasma Rifle
		 110 BFG 9000
		 111 Super Shotgun (DOOM 2 only)

      bit 6     unused
      bit 7     indicates a special action which alters the meanings
		  of the other bits:

		bits 1-0 = 01 pause or unpause
			 = 10 save game in slot # recorded in bits 4 to 2
				(slot number can thus be 0 to 7 but
				 should NOT be 6 or 7 or else!)

  There might be other special actions. The save game action happens
during replay of the demo, so be careful when playing demos if you
have important savegames! One or more of them could conceivably get
overwritten.

(3) The last byte of a demo has the value 128 (0x80)

edit: From the theoretical viewpoint it would be interesting to "crack" pre-v1.2 engine behavior the way entryway did with v1.2, but practical gains would be minimal. There are very few public v1.1 demos and v0.99 demos appear to be limited to the three lumps embeded in the IWAD. Still, it would be nice to have the differences documented.

[kb1]

Thanks for the details! As crazy as this demo looks in my port, I am starting to believe that maybe the speeds are being interpretted differently in 1.0 (0.99) (I'm sure I've seen it referred to with both numbers somewhere.)

Time for some serious DosBox visual comparisons! Or maybe some of our resident .exe hackers would care to take a look... :)

[Quasar]

kb1 said:

Thanks for the details! As crazy as this demo looks in my port, I am starting to believe that maybe the speeds are being interpretted differently in 1.0 (0.99) (I'm sure I've seen it referred to with both numbers somewhere.)

Time for some serious DosBox visual comparisons! Or maybe some of our resident .exe hackers would care to take a look... :)

Get me a link to the proper EXE and I'll load it up in IDA Pro.

[Never_Again]

v0.99
v1.1

Maes said:

I don't remember exactly how the raw demo lumps are parsed, but the cheating -or not- state is encoded in the player info

It is not. You must be thinking savegames, not demos.

[LogicDeLuxe]

I wonder, what is the point in savegame functionality in demos?

[myk]

The initially planned to allow recording from saves and included a -recordfrom parameter in earlier versions. The point to them would have been that demos could start in the middle of a level, perhaps making what level a demo is on less evident to newer users, or allow dramatic scenes where the player starts in a damaged condition or something.

Never_Again said:
First, let's clear up a misunderstanding invoked by the title of this thread: there's no DOOM v1.0.

Yeah there is, it's v0.99. Check out how the v1.1 manual says "note to those upgrading from 1.0 to 1.1". While the internal engine identifier equals 99, id also called it 1.0, otherwise, so people saying "v1.0" aren't (necessarily) talking about an inexistent version or misnaming the first release. If someone says something like "hey, there's a 0.99, but also a 1.0" then, yeah, they're wrong, but not if they're referring to the same thing.

[Csonicgo]

myk said:

.

Yeah there is, it's v0.99. Check out how the v1.1 manual says "note to those upgrading from 1.0 to 1.1". While the internal engine identifier equals 99, id also called it 1.0, otherwise, so people saying "v1.0" aren't (necessarily) talking about an inexistent version or misnaming the first release. If someone says something like "hey, there's a 0.99, but also a 1.0" then, yeah, they're wrong, but not if they're referring to the same thing.

He's just being a pedantic jackass. I don't understand why people get their jollies being so nit-picky.


What I wished happened in demos was that pause was never counted in the demo. Lots of demos by Cato has a shitload of pauses in them that I wish could be cut out.

[LogicDeLuxe]

myk said:

The initially planned to allow recording from saves and included a -recordfrom parameter in earlier versions.

Indeed. I remember. Yet, it feels not useful at all when it comes to spreading the demo. You would only annoy people by asking them to overwriting their savegames, and more so when the demo does this without warning. It would have been more useful, when the savegame actually could be part of the demo file.



Here is the tool btw.: http://www.doomworld.com/idgames/index.php?id=3428

[
* get information about one or more LMP files
* change the version of an existing LMP files
* change the player number who recorded the multi-player LMP file
* add some seconds of waiting at the end (good for COMPET-N files)
* remove the PAUSE parts
* decompiles (a part of) a LMP file to a Lmp Source (LS) file (ASCII text)
* compiles one or more LS files to a LMP file
* full source code included
* binaries for MS-DOS and LINUX (a.out) included

Should do all which you have asked for. I didn't try it, though. I'm just pointing to its existence.

[Never_Again]

myk said:

Yeah there is, it's v0.99. Check out how the v1.1 manual says "note to those upgrading from 1.0 to 1.1".

Yes, I'm aware of id's internal confusion about version numbers - cf. the v1.9 versioning mess. I guess it's a moot point whether the "internal" 0.99 number was correct and the later references to it as 1.0 merely revisionism on id's part.

edit:

Csonicgo said:

I don't understand why people get their jollies being so nit-picky.

No, you don't understand. What you may call nitpicking others may call "dotting your i's and crossing your t's". Ever consider that people who do that do it for the sake of signal-to-noise ratio? Not everyone comes here to "get their jollies", some actually want to find interesting, valid information that is useful. I'd rather be a well-informed pedant than a jackass who posts
1) "sky is falling" type of BS
2) questions that can be answered with two clicks via Google (LMPC and pauses in demos in this very thread)
3) just to vent (ibid)

It is better to keep your mouth closed and be thought a fool than open it and remove all doubt; but I guess with a 3000+ postcount that saying doesn't apply to you.

[Csonicgo]

Never_Again said:

It is better to keep your mouth closed and be thought a fool than open it and remove all doubt; but I guess with a 3000+ postcount that saying doesn't apply to you.

You really bite the trollbait, don't you? Thanks on derailing the thread even more, by the way! And that link of the "BS" (which you conveniently edited your post to blank it) actually spurred some discussion as to how unusable/hard-to-use doom files would be dealt with... so why is that BS?

LogicDeLuxe said:

Indeed. I remember. Yet, it feels not useful at all when it comes to spreading the demo. You would only annoy people by asking them to overwriting their savegames, and more so when the demo does this without warning. It would have been more useful, when the savegame actually could be part of the demo file.

This has always concerned me, and I think the reason they didn't is because the resulting file would be gigantic. Demos are already quite large in terms of 1993 space limits, and adding savegames (which are just snapshots of the internals as they are iirc) would bloat the demos quite a bit. That would cause some issues trying to share them. a few savegames would easily make 1 MB-3 MB demo files, which is just too large.

[Maes]

Csonicgo said:

Demos are already quite large in terms of 1993 space limits, and adding savegames (which are just snapshots of the internals as they are iirc) would bloat the demos quite a bit. That would cause some issues trying to share them. a few savegames would easily make 1 MB-3 MB demo files, which is just too large.

Sorry, but this is BS. The typical vanilla demo file is in the region of the few 10s of KBs, and modern source ports are pretty much in the same range. Vanilla savegames are kapped to a 100 or so KB tops anyway, which were small by 1993 standards (compared to the game itself, which could range from anything under just 5 to over 14 MB, and there were larger games still).

Yeah, they are a snapshot of the internals but the information they are storing is ridiculous even for 1993 standards. I could easily fit 20 savegames uncompressed on a floppy disk, and if I simply used PKZIP, both demos and savegames got mercilessly flattened because they contain so much redundant and repetitive data.

[LogicDeLuxe]

Maes said:

Yeah, they are a snapshot of the internals

I wonder, why vanilla has some problems with loading them, then. One of the more obvious effects are the monsters falling asleep. Are those just bugs in the restoring routine, or were there information actually left out? Maybe on purpose to keep them from growing even bigger?

[Maes]

It's a bad design decision/omission in the saving and restoring routine.
From void P_UnArchiveThinkers(void):

	  case tc_mobj:
	    PADSAVEP();
	    mobj = Z_Malloc (sizeof(*mobj), PU_LEVEL, NULL);
	    memcpy (mobj, save_p, sizeof(*mobj));
	    save_p += sizeof(*mobj);
	    mobj->state = &states[(int)mobj->state];
	    mobj->target = NULL;
	    if (mobj->player)

The target of mobjs (aka monsters in this case) is always set to null, even though the WHOLE contents of the mobj_t structure are saved to disk, including a pointer to their target.

The problem is that this pointer will only be valid within one game instance. Ending a game, purging stuff from memory and then recreating thinkers in an arbitrary order (they are part of a linked list that can go through multiple, detailed insertions/deletions at any time and no mobj is identified by a unique index or id or hash or whatever) makes all saved "pointer" invalid. They only indicate where in memory the pointer of the target was at the moment of saving. After loading, that same memory location can be full of garbage for all you know.

So yeah, they DO save "full" info even in vanilla savegames, only that the way it is done it's worthless. If mobj's had a unique index value to identify them, then it would make sense storing it.

In fact, Boom and its derivatives do just that:

        // killough 2/14/98: convert pointers into indices.
        // Fixes many savegame problems, by properly saving
        // target and tracer fields. Note: we store NULL if
        // the thinker pointed to by these fields is not a
        // mobj thinker.

[Quasar]

Maes said:

The problem is that this pointer will only be valid within one game instance.

Though, fraggle found it was actually in fact possible to reconstruct all mobjs' targets *from* the raw pointer values, because they are all unique. The reason this is possible is because the addresses of ALL mobjs are saved in the next or previous mobj when the thinker_t in their header is "accidentally" archived along with the mobj_t.

typedef struct thinker_s
{
  ...
  struct thinker_s *prev;
  struct thinker_s *next;
  ...
} thinker_t;
...
typedef struct mobj_s
{
  thinker_t thinker; // First in struct
  ...
} mobj_t;

Since thinker is at the head of mobj_t (necessary for DOOM's type punning code to work in the first place), and because the thinker list is saved in consecutive order starting at thinkercap.next and ending with the thinker having next = &thinkercap, the address of the "first" mobj is in the next mobj's prev pointer, etc.

The only gaps will be where other types of thinkers were in the list, but evidently there's generally enough info to determine those through process of elimination.

So id could have used this even if they didn't understand how to do pointer swizzling (and evidently they didn't). Kinda sad :P

[Maes]

Quasar said:

So id could have used this even if they didn't understand how to do pointer swizzling (and evidently they didn't). Kinda sad :P

Well, surely we can forgive them this little oversight/laziness in the greater scheme of Things (pun intended)?

[kb1]

Never_Again said:

...2) questions that can be answered with two clicks via Google (LMPC and pauses in demos in this very thread)...

Please describe these two clicks that would have answered my questions, or had even got me to LMPC...Cause, I'll click 'em.

I'm starting to think that maybe in 1.0, the movement speed bytes were read and scaled to represent slower speeds for a given number than with later versions.

It would make sense that, in v1.0, maximum speed was represented by a rather large byte value, but, when the turbo mode was to be added, this had to be scaled back to accomodate turbo ranges. It's just a guess that I will attempt to empirically verify this weekend or so.

[Quasar]

I'll get to IDAing the 0.99 executable on Thursday, btw. Then we'll have a definitive answer. All I have to do is find the G_WriteDemoTiccmd function - given the fact I've done the same sort of process multiple times (for Strife 1.0, Strife 1.2, Doom 1.3, Heretic 1.0, and the Press Release Pre-Beta), it shouldn't be difficult.

One just has to follow the calls from main -> D_DoomMain -> D_DoomLoop -> TryRunTics -> G_Ticker and from there you can find all the G_ functions easily.

[kb1]

Quasar said:

I'll get to IDAing the 0.99 executable on Thursday, btw.

I haven't found the time to try IDA extensively yet. I downloaded the free version, but didn't you mention at one time that you use a version that disassembles back to C code? Is that an add-on you had to buy, or does it come with the free version? Which version do you use? (Not trying to get you to do all the work, but you're probably better equipped ATM :)

[Never_Again]

kb1 said:

Please describe these two clicks that would have answered my questions, or had even got me to LMPC...Cause, I'll click 'em.

I don't understand what made you think that was directed at you. Are you Csonicgo? But since you asked, here you go.

BTW, anyone found any v0.99v1.0 demos other than those inside DOOM1.WAD?

[kb1]

Never_Again said:

I don't understand what made you think that was directed at you. Are you Csonicgo?

Oops, guess it wasn't.

Never_Again said:

I don't understand what made you think that was directed at you. Are you Csonicgo? But since you asked, here you go.

That was more than 2 clicks (for you)...and, required that you already know "LMPC"...

Never_Again said:

BTW, anyone found any v0.99v1.0 demos other than those inside DOOM1.WAD?

I haven't. For me, I simply want the IWAD to play correctly, as well as gain an understanding of the history of source modifications.
I significant problem is that there's no way to programmatically distinguish 1.0 vs. 1.2 demos, so the presence of the IWAD would have to be used to make the distinction.

BTW, how did this thread end up in Doom Speed Demos? It's clearly a source port thread, which, if I'm not mistaken, is where I started it.

[Grazza]

Your question is about the demo format (and thus very appropriate to the Demos forum), and is not really to do with source ports at all.

The people who frequent the demos forum are the most likely to know most about the demo format, and to think of ways to check and test this stuff, and they might miss this thread if it is in a different forum.

BTW, LMPC has its own sourceforge site. From the mess of tics in the sample cited, it looks like LMPC is misreading the data, rather than it simply being a matter of scaling. (Could check by recording demos in 0.99/1.0 and 1.1 where you do nothing but run/walk foward, or strafe, etc., and seeing how LMPC reads it.) I haven't found anything in Uwe Girlich's notes about <1.2 demos, but that doesn't mean he never investigated them. Anyone else remember anything about this?

kb1 said:

One significant problem is that there's no way to programmatically distinguish 1.0 vs. 1.2 demos, so the presence of the IWAD would have to be used to make the distinction.

Actually, given that the data format appears to have significant differences (tics that don't make sense if it is a 1.2 or later demo), I don't think that would be a major problem. After all, prboom+ autodetects Tasdoom demos (whose headers are 1.9 format) by spotting the data scrambling.

[Ryback]

From memory it is a scaling issue, since 1.1 will happily play 1.2 demos, but the player movement looks lethargic, like they don't know how to run.

I thought I'd test it with a simple run using just GF25 and GF50. Recorded under 1.2, the GF25 tics are recorded as 0x19, and the GF50 as 0x32 - as specified in the Unofficial Doom Specs.

The same demo recorded under 1.1, GF25 tics have the hexadecimal value 0x38, and GF50 is 0x71. A forward speed of 50 is being recorded as 113, so there must be some kind of scaling going on. You can see the acceleration demonstrated by playing the 1.1 demo back in the 1.2 engine.

Can't rule out LMPC not reading the 1.1 demo format properly though, since so far it refuses to recognise any of the 1.1 demos I've recorded as a valid lmp.

[Ryback]

Nevermind, LMPC is reading them now. Apparently it doesn't like it if your 1.2 format demo is too short. Anyway, same scaling applies to strafe tics - SR40 becomes SR91, SL24 becomes SL54. Interestingly there doesn't seem to be any scaling applied to turn tics - TL5/TR5 is the maximum value achievable with keyboard control, and it's the same value in a 1.1 demo as well.

[Donce]

This is not exactly on the original topic, but...

Grazza said:

After all, prboom+ autodetects Tasdoom demos (whose headers are 1.9 format) by spotting the data scrambling.

Actually, the first byte of a Tasdoom demo is ASCII character 'n' instead of 'm' for v1.9 demo, which indicates that Tasdoom is version "1.10". So there is no need to look at the data for detecting it's a Tasdoom demo. Now, to play it back - yes, you need to unscramble the data. I'm still surprised that I managed to make a semi-usable TAS->1.9 converter by comparing a converted vs. unconverted demo, given that I am almost totally null in programming.

Or did you perhaps mean that Prboom+ can detect a Tasdoom demo even if its version byte is changed to 1.9?

[kb1]

Ryback said:

From memory it is a scaling issue, since 1.1 will happily play 1.2 demos, but the player movement looks lethargic, like they don't know how to run.

Thanks, Ryback! I'll have to try it out with a 1.2-able source port. Don't quite expect it to stay in sync, but, it should be closer.

I imagine you could guestimate a detection of an old demo based on these values, but it would sometimes mistake a 1.2 demo for a 1.0/1.1 if the 1.2 demo was recorded with -turbo. Probably a very rare occurance indeed, maybe too rare to even worry about. Then again, 1.0/1.1 demos are also rare, so maybe it's best to leave that task to the user.

[Never_Again]

kb1 said:

I imagine you could guestimate a detection of an old demo based on these values, but it would sometimes mistake a 1.2 demo for a 1.0/1.1 if the 1.2 demo was recorded with -turbo. Probably a very rare occurance indeed

Never_Again said:

Given that -turbo didn't exist in v1.2 and earlier