The wonderful world of -statcopy

[fraggle]

Vanilla Doom has a mysterious command line parameter named "statcopy". The Doom FAQ describes it as "Unknown". What does it do? Read on..

Doom comes with ipxsetup.exe and sersetup.exe, external programs that Doom communicates with to run multiplayer games over IPX networks and serial/modem lines. You run ipxsetup, and it runs Doom for you, adding a magic extra "-net" option that you never see. The -net option is used to specify a memory address that doom.exe and ipxsetup.exe use to communicate.

The "-statcopy" parameter is similar to "-net": it's used to specify a buffer to pass the end-of-level statistics into. It's supposed to be passed by an external statistics-gathering program to capture the intermission screen statistics, just like how ipxsetup and sersetup pass "-net". The only problem is that no such program has ever been written (at least, until now).

Over the past few days I've written what is, as far as I know, the world's first Doom external statistics driver. You can download it here.



My driver just saves the statistics to a file, which isn't very exciting. However, you could do more interesting things with it. For example:

• Use the statistics data to generate demo hash checksums, and automatically check that source ports play back demos exactly the same as Vanilla Doom.
  
• At a Doom tournament, pass the statistics data out in real time to some kind of big screen that shows players' scores.

The real mystery is why Id added this feature into Doom, though. However, it appeared in v1.666, which is the same time that the "-avg" "Austin Virtual Gaming" parameter appeared. Maybe it's something to do with the Austin Virtual Gaming tournament? (See my tournament / big screen idea above for an example of how it might have been used)

[exp(x)]

You are awesome.

[RazTK]

exp(x) said:

You are awesome.

What I was about to say. :>

[myk]

Sounds quite cool, but following the instructions I can't get it to output any stats. It always says "Statistics captured for 0 levels". I warped to level 1, reached the exit, entered level 2, and quit. It does create a STATS.TXT, but it's correspondingly empty. An example of one of the (simplest) command lines I tried was:

d:\doom\statdump doom2 -warp 01 -skill 4 

[RTC_Marine]

God has graced us once again, all hail fraggle!

[fraggle]

myk said:

Sounds quite cool, but following the instructions I can't get it to output any stats. It always says "Statistics captured for 0 levels". I warped to level 1, reached the exit, entered level 2, and quit. It does create a STATS.TXT, but it's correspondingly empty. An example of one of the (simplest) command lines I tried was:

d:\doom\statdump doom2 -warp 01 -skill 4 

Do you have use_mouse disabled in your Doom config? If so, try turning it on.

And yes, you understood correctly: this only works if you have the mouse enabled. Really.

[myk]

I do use the mouse to play (why is it required?). I disabled novert just in case that was causing this, but it still didn't give any stats. I'm on Windows 98, in case that matters.

Are there any other settings that, like mouse usage, may cause it to output no stats?

[fraggle]

myk said:

(why is it required?)

In order to capture the statistics for each level it's necessary to periodically check the statistics buffer to see if Doom has written stats for a new level. The normal way to do this would be to set up a timer interrupt to run every few seconds and check.

However, Doom installs its own timer interrupt handler, so if I did that it would just be overwritten. The only solution I could come up with in the end was to replace the mouse driver interrupt. Every time Doom calls into the mouse driver, it's actually calling into statdump, which checks the buffer and then calls the real mouse driver.

Probably the ugliest hack ever known to mankind, and it means that it won't work if you have use_mouse turned off, but I couldn't see any other way.

. I disabled novert just in case that was causing this, but it still didn't give any stats. I'm on Windows 98, in case that matters.

Are there any other settings that, like mouse usage, may cause it to output no stats?

Mouse usage shouldn't make a difference. I haven't tried in Windows 98 (only tried in DOSBox), so you've probably found a bug.

[fraggle]

Just tried in Windows XP, and I get a nasty crash when I reach the intermission screen. Ouch. Looks like I need to test this more.

[myk]

fraggle said:
Just tried in Windows XP, and I get a nasty crash when I reach the intermission screen. Ouch. Looks like I need to test this more.

I got one on Windows 98 too, but only after I opened a clean "default" prompt to test with. It gave a terminal error 06, or something like that each time, and Windows couldn't recover from it. Perhaps my usual DOOM command prompt ignores the issue (maybe also the stats) because it's set to "prevent MS-DOS programs from detecting Windows".

[fraggle]

I found the bug. I just uploaded a fixed version :-)

[myk]

On my end the "ignore Windows" prompt is still giving stats for 0 levels, and the default prompt still crashed, but instead of giving the BSOD with the error message it just totally froze on the ingame screen* just when I hit the exit switch.

* The player hitting the switch.

[fraggle]

myk said:

On my end the "ignore Windows" prompt is still giving stats for 0 levels, and the default prompt still crashed, but instead of giving the BSOD with the error message it just totally froze on the ingame screen* just when I hit the exit switch.

* The player hitting the switch.

Drat. I'd like to fix it but I only have XP and Dosbox to test with, unfortunately.

:-/

[myk]

fraggle said:
I only have XP and Dosbox to test with, unfortunately.

Maybe if some other coder has DOS or Windows 9x he could do some debugging on the source...

[entryway]

fraggle said:

However, Doom installs its own timer interrupt handler, so if I did that it would just be overwritten. The only solution I could come up with in the end was to replace the mouse driver interrupt.

How about the second timer (rtc int70?)? Also, you are not obliged to use mouse or int70 interrupt permanently. Only do check there and if your int8 was stolen, then reinstall it and kill himself heh (I mean the second guard interrupt)

[Kaiser]

This is why the Doom community lives on.

[fraggle]

One suggestion that's been made is that statcopy might have been added for DWANGO. This is certainly an interesting theory that warrants further investigation. I tried running strings on dwango.exe but I don't get any useful output. It looks like the exe is compressed.

[entryway]

Try to finish the first level from doom2.wad with such command line
DOOM95.EXE -warp 1 -skill 4 -nodm -statcopy 4727884

[myk]

Weird, is that a memory overflow? And after level 1 in loads level 28.

Fraggle, isn't it possible for you to build some sort of debugging version people on DOS or Windows 9x could use to output data for you to catch the bug(s)?

[entryway]

myk said:

Weird, is that a memory overflow? And after level 1 in loads level 28.

It rewrites wminfo.next field with value from wminfo.maxkill field.

4727884 is a address of (wminfo-4). statcopy param force doom to do the following copying in G_DoCompleted()
memcpy (statcopy, &wminfo, sizeof(wminfo)); // to, from, size

So if statcopy equal to wminfo-4, then all values of wbstartstruct_t struct will be shifted up by 4 after copying:
wminfo.didsecret -> wminfo.epsd
wminfo.last -> wminfo.didsecret
wminfo.next -> wminfo.last
wminfo.maxkills -> wminfo.next
etc

maxkill for map01 is 27 (number of monsters). Hence after copying wminfo.next will be equal to 27 and the next level will be wminfo.next+1=28

The same address for DOS exe is 1891756 (at least for XP)

Strange things can be done by means of -statcopy

[entryway]

http://prboom-plus.sourceforge.net/speed.zip

It's my small movie of map01-map05 in 22 seconds. To play this demo you should use DOS exe (maybe in XP from dosbox only) with following command line:
doom2.exe -playdemo speed -statcopy 1892116

[RazTK]

AMAZING! :D

EDIT: Yay for genuine noclip hack!

[entryway]

EDIT: Yay for genuine noclip hack!

How do you start it? It works for me only from dosbox 0.71

[RazTK]

I'm using DOSBox too, it hangs when you execute it with the same command-line without DOSBox.

EDIT: Are you using a debugger to find the addresses? If so, what debugger are you using? I'd also like to experiment with it a little. :>

[entryway]

RazTK said:

EDIT: Are you using a debugger to find the addresses? If so, what debugger are you using? I'd also like to experiment with it a little. :>

I used fully disassembled by me doom2.exe and doom95.exe with IDAPro, Hiew and piece of my experience for getting runtime addresses of "some interesting variables" for DOS exe. For doom95 they are static and can be seen in IDA directly. For DOS exe I need an ImageBase for calculation of real addresses. For DOSBox it equal to 0x133000.

Look at this:
http://prboom-plus.sourceforge.net/ida.png

0x133000 + 0x9adb0 = 1891760 - address of wminfo
(wminfo-4) = (1891760-4) = 1891756

entryway said:

The same address for DOS exe is 1891756 (at least for XP)

EDIT: I don't know how can I debug DOS doom2.exe, but doom95.exe works perfectly with IDAPro built-in debugger

[RazTK]

Meh, if we just had some sort of program that would show you what address is being executed/changed in real-time, such things could be way more easy. But maybe I'm just bullshitting here. :>

EDIT: In VisualBoy Advance, for example, when you want to find the address of which your health is stored at, the emulator can search for all the addresses that holds the value "100", and if it finds more than one, you would just keep playing and when your health changed, you could see what address has been changed and find the correct one. But I guess it's working in a whole different way, I'm not sure.

[entryway]

RazTK said:

EDIT: In VisualBoy Advance, for example, when you want to find the address of which your health is stored at, the emulator can search for all the addresses that holds the value "100", and if it finds more than one, you would just keep playing and when your health changed, you could see what address has been changed and find the correct one.

I saw the similar program for DOS long long ago.

[VinceDSS]

would -statcopy work for stats in coop games ?

[entryway]

yes, at least by design
wminfo contains stats for all players

[VinceDSS]

in the case of a recorded max coop game, would it report the infight kills as well ?