Chocolate Strife Beta 1 now available

:/ Is the existing version not functional enough to do anything with?

In other news I've stumbled across yet more undefined behavior in the executable.

In P_XYMovement a modification was made to allow missiles to activate shoot-type (G1/GR) lines that they impact. It does this first by checking the "blockingline" variable which Rogue added that tracks the linedef that stops movement in PIT_CheckLine. But then it does something very iffy:

  if(numspechit)
     P_ShootSpecialLine(actor, spechit[numspechit-1]);

This is all fine and great if spechit is greater than 0. Unfortunately P_Move is known to regularly leave numspechit equal to -1. BOOM had a problem with this... and so does Strife. The code as written in P_XYMovement is capable of accessing spechit[-2]. While vanilla Strife appears to tolerate this access, Chocolate is crashing at random due to it.

I've reworked the sound buffer allocation system. Cached sounds are now no longer stored on the zone heap. There are minor things I still need to finish off but it should fix the out-of-memory errors and remove the need for the -mb parameter.

fraggle said:

I've reworked the sound buffer allocation system. Cached sounds are now no longer stored on the zone heap. There are minor things I still need to finish off but it should fix the out-of-memory errors and remove the need for the -mb parameter.

Good news! Better news than mine, for sure ;) I'm waiting on entryway to see if he knows anything about the contents of DOS memory at offset 0x12 :P

EDIT: Well I didn't get an answer from entryway but I did manage to find a dump of the DOS 6.22 interrupt table. Offset 0x12 in that version at least is 0x0070, or 112, which isn't a valid shoot line type. I am willing to bet it just happens to not be a valid shoot line type in other versions as well, or this would blow up very badly I think. I'm going to emulate the observed behavior by changing the condition to numspechit > 0.

entryway said:

Not really a problem. If you want to address some own data or get access for internal variables without adding fixup you can use call $ pop eax (e8 00 00 00 00 58) for getting real address. See doom2-plus.crk "saves" section:

00086F94: 00 E8 ;call        000061D85
00086F99: 00 59 ;pop         ecx
00086F9A: 00 83 ;sub         ecx,9

Does that work reliably if you want to access data or code in another section (=LE object)? I think the LE loader can relocate each object to pretty much anywhere in the memory (not right after each other) if it wanted to. Like object #0 @ 0x131000, object #1 @ 0x274900 and object #2 @ 0x392000 for example (sizes of each object being much smaller than the difference in relocated base addresses). Not sure if that ever happens in practice with DOS4GW even if it was possible in theory, though.

(I guess you could read one of the data references from inside the code section and then calculate the data section's relocated base address from it... or, if you're putting code inside the data section, read a function pointer and use it...)

When I got tired of fiddling with the fixups manually all the time I just made some tools that regenerate the whole fixup record table when needed. :-) It's not totally automated yet, but at least I can edit them in a handy, readable format (it's a text file and uses linear addresses instead of page-based and source/target object numbers, 16 vs. 32 bit fixups and fixups that cross page boundaries are all handled automatically). Not sure if that has the potential to cause problems though (haven't had any myself). If nothing else, at least it'll make patch files bigger because in the original executable the fixup records are not always in a strictly ascending order based on the source address, unlike after the table has been rebuilt.

Quasar said:

Good news! Better news than mine, for sure ;) I'm waiting on entryway to see if he knows anything about the contents of DOS memory at offset 0x12 :P

I think you can get it easily in such way (for doom binary):

cseg02:0003915F push esi
cseg02:00039160 push offset aP_giveammoBadT ; "P_GiveAmmo: bad type %i"
cseg02:00039165 call I_Error

Injected code (you can inject it at start of P_GiveAmmo for example):
mov esi, [12]
jmp 0003915F

Also I suggest you to replace "P_GiveAmmo: bad type %i" with "P_GiveAmmo: bad type %x" :)

xttl said:

...When I got tired of fiddling with the fixups manually all the time I just made some tools that regenerate the whole fixup record table when needed. :-)...

Interesting idea. I could never exactly figure out the fixup format - it looked like the addresses were not 4-byte addresses, but rather, offsets or something. There must be another table somewhere that is used to add the base to each fixup to calculate the exact address. For my strife hack work, I had to "borrow" fixups that pointed "near" the actual location I wanted to fixup, otherwise it wouldn't work.

kb1 said:

Interesting idea. I could never exactly figure out the fixup format - it looked like the addresses were not 4-byte addresses, but rather, offsets or something. There must be another table somewhere that is used to add the base to each fixup to calculate the exact address. For my strife hack work, I had to "borrow" fixups that pointed "near" the actual location I wanted to fixup, otherwise it wouldn't work.

The fixup records are stored separately for each page (pages are 4kB) and the source address (the location to be patched when loading the LE) is always given as an offset relative to the start of that particular page. Whenever you have to add a fixup record for an address that crosses a page boundary, you must have 2 records: one for each page. The offset for the 2nd page's record is then negative.

Destination addresses are given as a destination object number and then an offset inside that object, but each fixup record has a flags byte which you can use to toggle between a 16-bit and a 32-bit destination address. So if your fixup's target is inside the first 64kB of a particular object, you can leave the 32-bit flag unset and save 2 bytes that way.

I can give you the source for my fixup dumper/rebuilder tool if you want, though it's a giant mess because I am kinda lazy and I already got it working for all the executables I care about (that's all versions of Doom/Strife/Heretic/Hexen, including pre-release versions of Doom). :-D The output from it looks like this: http://taateli.fi/rnd/newdoom-original-fixups.txt (those are from an unmodified Doom press beta executable). You can then edit that list in a text editor and regenerate the whole fixup record table automatically from it. It's not possible to increase the total size of the fixup record table yet (sizes for each individual page's part of the table don't matter though, and decreased total size is ok), but all of the Doom-related executables have lots of unused code and data or a gazillion inline copies of a particular function so it's not that big of a deal IMO. The list is also automatically sorted so you can keep your own mods near the top of the file.

Right now you have to keep editing that list manually, but it's of course possible to make nasm output an object file instead of a flat binary (I usually use nasm for bigger hacks, I take a particular function from the disassembly and edit it so that it can be assembled with nasm and inserted back into the executable) and then use the additional information in the object file to output fixup record lines in the appropriate format automatically. I already have that sort of working, but I've been too lazy to finish it properly. (...or the whole Doom 0.8 thing for that matter, haha)

Btw. this can also be useful: ftp://ftp.tenberry.com/ftp/pub/lxexe.doc.

xttl said:

The fixup records are stored separately for each page (pages are 4kB) and the source address (the location to be patched when loading the LE) is always given as an offset relative to the start of that particular page. Whenever you have to add a fixup record for an address that crosses a page boundary, you must have 2 records: one for each page. The offset for the 2nd page's record is then negative.

But, where's the page address to add to the offset? In other words, with strife1.exe, I extracted the LE, and, in desperation, found the fixups table. I then tried to write a program that would read the fixups table, and give me the offsets and function/varnames. Here's an excerpt:

000016D9: 07 10 DE 06 03 AC F1 02 00    cseg01:0001A6DE  ->  dseg03:0009F1AC [gameaction]
000016E2: 07 10 D5 06 03 9C F1 02 00    cseg01:0001A6D5  ->  dseg03:0009F19C [paused]
000016EB: 07 10 CF 06 03 BC C1 02 00    cseg01:0001A6CF  ->  dseg03:0009C1BC [demosequence]

The "cseg01:########" addresses are only correct for some of the lines listed - for example I took the first line, read the DE 06 03, read it as Segment 3, address 000006DE, and added 0001A000 to arrive at "cseg01:0001A6DE" which (I think) is correct for that particular entry. But, it's not correct for every fixup, just one's that are nearby - in the same page, I suppose.

So, my question is "How do I know that I need to add 0001A000 to this particular line? How do I know which page corresponds to that particular fixup, and what is that page's address?" You may have explained it already, but I didn't understand.

BTW I didn't know what the 07 did, but I knew that if a 10 followed it, I got bigger addresses than if a 00 followed it. :)

xttl said:

I can give you the source for my fixup dumper/rebuilder tool if you want, though it's a giant mess because I am kinda lazy and I already got it working for all the executables I care about (that's all versions of Doom/Strife/Heretic/Hexen, including pre-release versions of Doom). :-D

Please do! That may just explain it for me! I am also *very* interested in the Doom 0.8 code and program - even as-is (but I understand if you're not ready to release it yet)

xttl said:

...I usually use nasm for bigger hacks, I take a particular function from the disassembly and edit it so that it can be assembled with nasm and inserted back into the executable...

Geez, I wrote my whole Strife patch with hand-written machine code - ouch! Using nasm is a great idea!

kb1 said:

So, my question is "How do I know that I need to add 0001A000 to this particular line? How do I know which page corresponds to that particular fixup, and what is that page's address?" You may have explained it already, but I didn't understand.

The LE header has info on how many pages in total are stored in the file, how many pages each object has and also what is the base address (unrelocated) for each object. Here are some fields from the header of Strife v1.31:

le_page_size..........................: 4096
le_num_pages..........................: 118
le_num_objects........................: 3


 obj#: vsize       base        flags       ptidx  nump  unk1
 -------------------------------------------------------------------
   00: 0x000497fe  0x00010000  0x00002005  1      74    0x00000000
   01: 0x00000019  0x00060000  0x00001005  75     1     0x00000000
   02: 0x00080cf0  0x00070000  0x00002003  76     43    0x00000000

So the file contains 118 pages of which 74 belong to the first object (code section) and 43 belong to the third object (data section). There's also one small object between them that has just one page associated with it, I don't know what's it about but all of the later LE files I've seen have it (Doom beta does not). Uninitialized data comes after the initialized data (and of course doesn't need any pages stored in the file) so it's not the BSS section. You can see this object in IDA as cseg02.

If you want to convert a page-based fixup source address to a linear address, first you have to find out which page's fixups you're dealing with. The LE header has a fixup record page table right before the actual fixup record table. It has one 32-bit offset into the fixup record table for each page stored in the file. If a particular page has no fixups associated with it, then the last offset is repeated. After the final page's offset, the table is terminated with 0x00000000.

Using the info from the LE header, you can calculate which object any particular page belongs to and also get the linear base addresses for each object. Then you take the current object's base address, add pagenum*pagesize to it and finally add the page-based source address from your fixup record to it (count pagenum from that object's first page, not from page 0 unless of course you're dealing with the first object in that file).

kb1 said:

BTW I didn't know what the 07 did, but I knew that if a 10 followed it, I got bigger addresses than if a 00 followed it. :)

The first byte (usually 0x07 as you said) is the fixup record type and the second byte stores the flags for that fixup.

Almost all of the fixup records in Doom-related executables are type 7 and those few which aren't are type 5. I think type 5 means that the loader only patches 2 bytes and not 4 when it's applying the fixups, but I'm not sure about that. The only place you see type 5 fixups is inside that mysterious cseg02 or, in the Doom beta, a couple of times near the beginning of the first object (code section).

kb1 said:

Please do! That may just explain it for me! I am also *very* interested in the Doom 0.8 code and program - even as-is (but I understand if you're not ready to release it yet)

Ok, I'll send you a PM later. (so as not to derail this thread even further and because some of the stuff is kinda embarassing ;-) There's not even really a single executable that contains all the hacks for Doom 0.8, instead I have 3 or 4 different executables where I'm doing all kinds of tests, some random .wad, .lmp, .c and .asm files and so on.

kb1 said:

Geez, I wrote my whole Strife patch with hand-written machine code - ouch! Using nasm is a great idea!

I probably also have to make a tool which can automatically convert ndisasm's output or copy+paste from IDA to a re-assemblable form sooner or later... it's kinda annoying and error-prone to do it by hand for some of the bigger functions. :-(

edit: fixed typo.

This will seem OT after all the reverse engineering going on here, but a work-around for the Win7 palette issue seems to be to leave the screen resolution dialog open. If that doesn't work I have heard closing Explorer works. It leaves me at a loss, but it works in Chocolate Strife and Red Alert.

The current version of strife-branch has merged the latest changes from Chocolate Doom trunk, which include a fix for the palette issue, so once beta 2 is released the problem should hopefully go away.

If you can try Chocolate Doom 1.5.0 (download is available on the website) and can confirm that you don't get the problem (ie. that the fix works for you), it would be helpful.

Hopefully beta 2 should be released soon ... :)