Shadowcaster modding 2: the remoddening

[Gez]

Old-timers may remember this ancient thread. They'll also remember that nothing came out of it.

I'm still interested by the prospect, however, and have started a little hack-job with SLADE3 to at least identify and extract the lumps. Currently I can extract the content of SHADOW.LIB (both CD and floppy version) and SHADOW2.LIB. Which doesn't mean these lumps' own formats are understood, but that's still a start.

RAVDATA.DAT (along with many other .DAT files, especially in the CD version), however, uses a different archive format. I have not identified yet where the graphical data (including the palette) are kept. I would also be interested if anyone has any pointer for where in RAVEN.EXE are the actor tables located (to grab values like hit points, speed, XP value, etc.).

So if anybody has some technical expertise to offer, I'd be grateful. Thanks.

[Gez]

So it seems interest in this has kinda died. Too bad.

[AveryMaurice]

I think all the work you've done is quite impressive so far, I wish you luck on the project.

I myself do not really see the good in making levels for or modding shadowcaster. The game doesn't seem to have a very big community anymore and the actual level structure itself is quite simple.

[Gez]

At the moment, the thing that interests me the most is data extraction. Finding where the sprites are, how they are read, so that they can be extracted and converted.

[Gez]

I made great progress. Identified the palette and several HUD sprites. Also delineated what must logically be the world sprites I suppose.

[Gez]

Okay. It's a bit crude and hacky (very few sanity checks) but I don't care too much at the moment. For all zero people out there interested, I have a patch for Slade (apply to r494 and compile) that allows opening SHADOW.LIB (both floppy and CD versions), SHADOW2.LIB, RAVDATA.DAT, CD_CASTR.DAT and HD_CASTR.DAT.

Shadowcaster's palette is in 6-bit format (max intensity for a color channel is 63, not 255). So before it can be used in Slade to view the graphics, it needs to be converted to 8-bit format. It's quite simple and there's a right-click function to do that to any lump. (No sanity check again. :p)

The palette in HD_CASTR.DAT is in a completely different format from that in RAVDATA.DAT; though, and it starts with RLE0 which may be a coincidence or may mean that the palette is compressed. (If so, the size shrunk from 768 bytes to 752. Big whoop. Especially for the CD version of a game...)

Most HUD graphics are in a format that Slade already identifies, since it was also used in some Doom alpha versions. A minimal header for width and length, followed by a raw dump, row-major.

[kristus]

Cool stuff.

[Quasar]

Gez said:

Most HUD graphics are in a format that Slade already identifies, since it was also used in some Doom alpha versions. A minimal header for width and length, followed by a raw dump, row-major.

Ha, I knew it! >.<

I really wish I could help with this project. I could load up the EXE in IDA Pro, but without knowing for sure if there are correlations with the Wolf, RoTT, or Doom sources, I may not get anywhere. Have you thought about running strings on EXE to see what type of error messages etc. are in there? Might be useful to get a hunch about what kind of code is in there.

[Gez]

Quasar said:

Have you thought about running strings on EXE to see what type of error messages etc. are in there?

Here are the output of strings on both versions of the exe:
http://pastebin.com/8kFbAT4d

Relevant messages start at line 2379 for the floppy version. The CD version starts at line 3029 and relevant messages start at line 5568.

Edit: Progress!

This is pretty much the smallest, simplest sprite. Now that I've identified it (19f in ravdata.dat or hd_castr.dat) it should be relatively straightforward to extrapolate the format of its header. It seems pretty similar to the Doom picture format actually. One thing is sure, it's column-major and the first bytes are not graphical in nature.

[Quasar]

Early pass with IDA Pro and use of error messages and some basic examination got me the following symbols. This is a very small percentage in terms of coverage, however. The EXE is nearly as large as the DOOM prebeta in terms of both code content and data.

Some interesting things:

• The CA_ functions are ancestral versions of DOOM w_wad.c
  
• DrawSpans suggests floor rendering might share some of DOOM's approach to drawing flats
  
• INT_TimerISR is called I_TimerISR in DOOM, but the contents of the function are different because the gritty details are pushed off to DMX's TSR module in DOOM. This EXE does not use DMX.
  
• DrawPic is not quite the same as V_DrawPic in the DOOM prebeta, but it's hard to tell what is algorithmic and what is simply codegen differences, as this was compiled with a different version of Watcom.
  

 Start         Length     Name                   Class
 0001:00000000 000046D4AH cseg01                 CODE
 0002:00000000 000076690H dseg02                 STACK


  Address         Publics by Value

 0001:00001878       main
 0001:0000344E       HandleFace
 0001:00003771       SetOVImage
 0001:00005900       HandleSwing
 0001:0000C590       AnimateMonster
 0001:000103A1       GetSprite
 0001:00010CA8       RF_CheckActionFlag
 0001:00010D07       RF_RenderView
 0001:00011549       DrawSpans
 0001:00011BF4       RenderDoor
 0001:00018C66       CA_ReadFile
 0001:00018CD7       CA_LoadFile
 0001:00018EE7       CA_NumForName
 0001:00018F5B       CA_GetNamedNum
 0001:00018FA7       CA_LumpPointer
 0001:00019091       CA_ReadLump
 0001:000190FD       CA_FreeLump
 0001:00019183       CA_WriteLump
 0001:0001921F       OpenDebugLog
 0001:00019248       CloseDebugLog
 0001:000194D5       INT_TimerISR
 0001:000197B7       StartupTimer
 0001:00019946       Error
 0001:0001A3BE       DrawPic
 0001:0001A475       MaskDrawPic
 0001:0001A551       DrawSubPic
 0001:0001A7DA       MaskDrawSubPic
 0001:0001A9B4       DrawPicR
 0001:0001AA7C       DrawSubPicR
 0001:0001ADA3       DrawShadowedPic
 0001:0001B311       SetUnderImage
 0001:0001CC4F       CreateItem
 0001:0001DA8F       SlideObject
 0001:0001DFF8       SetPage
 0001:0001F94B       UpdateHS
 0001:00020FD2       KeyWalkOn
 0001:000210E0       KeyWalkOff
 0001:000211DD       KeyFlyOn
 0001:000212F6       KeyFlyOff
 0001:00023F17       hsChangeViewSize
 0001:00030E1F       DoCastSpell
 0001:00031EEF       MCastSpell
 0001:00033BF6       LoadArchtypes
 0001:000341C5       LoadMonsters
 0001:00034804       LoadItems
 0001:00034AE7       LoadCellItems
 0001:00034D00       LoadCellDoors
 0001:00034EAF       LoadCellObjs
 0001:00035B26       InitSound
 0001:00036681       LoadSample
 0001:00036D77       LoadSong
 0001:0003CCC9       LoadGame
 0001:0003D9C1       SaveGame
 0002:0004A958       debuglogfile

Program entry point at 0001:0003ECCA

[Gez]

Some more fooling around with that sprite.


Here is its original code. Underlined are known pixel data (they are the values I changed to change the colors).

02 00 05 00 0E 00 11 00 1F 00 2D 00 3B 00 09 09
FF 0B 00 33 38 0D FF 4B 4B 47 4B 4E 4B 50 01 0B
00 3D 3F 08 0A 4E 4F 4F 4F 50 51 51 01 0B 00 38
3D 0A 0C 4B 47 4B 4B 4B 4E 50 01 09 09 0C

0B corresponds to the length of the following column. When I changed all three of them to 08, I got a shorter potion. 00 might be its vertical offset? When I changed that value I got a divide by zero crash.

Pixel value 00 is transparent.

Format does not care whether columns overlap:

11 00, 1F 00 and 2D 00 appear to be the offsets of the column info start. 0E and 3B are offsets to 0909 pairs. Hmm. Not sure what 02 and 05 mean.

Identified another item sprite. 5f. It's the fire wand. That one is made of 19 columns of 1 single pixel each.

0A 00 14 00 2C 00 30 00 34 00 38 00 3C 00 40 00
44 00 48 00 4C 00 50 00 54 00 58 00 5C 00 60 00
64 00 68 00 6C 00 70 00 74 00 78 00 01 00 3A 01
01 00 7C 01 01 00 7F 01 01 00 7C 01 01 00 7F 01
01 00 0A 01 01 00 0B 01 01 00 0C 01 01 00 0D 01
01 00 0E 01 01 00 0F 01 01 00 10 01 01 00 0E 01
01 00 0D 01 01 00 0C 01 01 00 0C 01 01 00 0B 01
01 00 11 01 01 00 0F 01 01 01 0C

[Quasar]

Very poor correlation with the RoTT source so far, so I doubt it's going to be very useful. I did find one thing in it so far, however.

It has a function called CA_RLEWexpand which expects data to start with varying headers depending on the point of call, one of which can be "RLE". It is possible that this is the function used to decompress resources with RLE compression in Shadowcaster as well. It can be found in the RoTT source code in RT_TED.C at line 1265.

Edit: Wolf3D also has that function, but its version is in assembly code.

There is a strong correlation between the sub at cseg01:00026ADE and R_DrawColumn in the DOOM prebeta version. Both call into extremely large arrays of code pointers that execute a varying amount of mov/add/xor/shld instructions based on the length of the column being drawn.

[Gez]

I'm using a dumb, brute force approach to sprite reading: I start at offset 4-5 and keep reading the values as offsets to a column as long as they're in increasing order (which is technically not actually a requirement) and seem to be indeed offsets (do not point outside of the boundaries of the lump).

It... works. To an extent. There are some ugly artifacts at the bottom of some columns; and sometimes they oddly repeat. I'll need to refine the code, but there are so many control values who have no obvious effect.
Here are a few results.


Update: Found it. Solved. Sprites are loaded correctly now!
Specs:
The purpose of the first two bytes is still unknown.
The next two bytes describe the number of columns.
Then you have column offsets on two bytes each. If the offset is null, then that column is empty and must be skipped.
Then the columns themselves.

A column's first byte is its vertical offset (how far from the bottom it starts), the second byte is how far from the bottom it stops. So, like in Doom, it's possible that there may be pixel data left undescribed (above and below a column); if so it should be transparent. A column is solid, however, there may not be several posts per column like in Doom. To have transparent areas vertically in between visible pixels, index 0 is used as a transparent color.

[Quasar]

Great work. Sorry I couldn't be of more help >_> If there's anything in particular you'd like me to look at, just let me know.

EDIT: New version of the .map with some more symbols filled in, though most of the names are speculative based on what I have determined they are doing:

http://eternity.mancubus.net/text/raven-alt.map

There isn't that much correlation with Wolf3D in here either, frankly. There are parts that are the same, but the majority seems to be a fresh game engine implemented on top of the basic framework.

Probably shouldn't be too surprising considering how non-reusable most of the Wolf3D code was. I find it especially interesting that Shadowcaster has a scripting system.

[Gez]

There seems to be somewhere code for morphing a sprite into another. It can be seen in some transformation scenes, such as the impaled skeletons into skeleton warriors, the headless king into skeleton warrior, or the final boss fight. Because I did not find sprites for such animations (whereas there are graphics for player's own transformations, shown in the HUD).

[40oz]

thats it? you just punch it until it dies, making no effort to move away from its attacks?

[Snarboo]

This is really impressive Gez! Any chance you'll talk to Slayer to have this implemented into SLADE?

[Gez]

40oz said:

thats it? you just punch it until it dies, making no effort to move away from its attacks?

If you've "grinded" the Grost form enough, you don't need to move away. That form is very tough. Which is good because it's also very slow. You'll notice the use of earthquake spells, too, though.

If you try attacking him with a different form, it's a lot more challenging. Keep in mind though that that game has rather clumsy controls compared to Doom. So most of the fights are full-frontal bashing like that. (Exactly what Romero wanted Quake to be, heh.)

Snarboo said:

This is really impressive Gez! Any chance you'll talk to Slayer to have this implemented into SLADE?

Well, he seemed more open to the idea than I was about it. That means I'll have to clean up the code! :p (And probably not make these archive types read-only. At the moment, opening them is implemented, but writing such type of archive is not.)

[Snarboo]

Awesome! :D Any chance you'll consider doing Cyclones next? I think that's the only other Raven game which hasn't had its graphics ripped yet.

[Gez]

I do not have CyClones, so it'll be difficult. :)

[Quasar]

40oz said:

thats it? you just punch it until it dies, making no effort to move away from its attacks?

          

[Gez]

Quasar said:

Great work. Sorry I couldn't be of more help >_> If there's anything in particular you'd like me to look at, just let me know.

Any idea where the mobj tables are located? With hit points, XP award, and the like? I wonder if there's any way we could find out the stats for the critters and weapons.

[Csonicgo]

Great work Gez! now the T667 Beastiary and weapons mods will rip from this game too! ;)

[Gez]

The monsters have the same problem they have in Wolfenstein: no rotations for the attack frames. The weapons are... Well, they're not Doomy in their behavior. No idle frame. A rather bland attack animation for melee weapons (mostly a variety of weird fists and tentacles). No attack frames for ranged weapons (other than projectile sprites). I think we're more or less safe. :p

[Quasar]

Gez said:

Any idea where the mobj tables are located? With hit points, XP award, and the like? I wonder if there's any way we could find out the stats for the critters and weapons.

Not yet but I can dig deeper and keep an eye out for it.

Gez said:

The monsters have the same problem they have in Wolfenstein: no rotations for the attack frames.

Kind of a shame, because this game's art style is very similar to Heretic's. I think the monsters would fit in that game quite well. Not so much with DOOM. Heretic could always use some more love in the modding department.

[40oz]

Quasar said:

protip

That's it? You just shoot at it until it dies?

[Quasar]

Bad news / good news, depending on whether you value simply coming to an understanding of this game engine over actually being able to freely modify it.

Monster ID #'s are assigned particular AI types via some kind of data that is loaded either from static data in the level or in its script. I have found the array of script action routine pointers, and I have found code that loads and allocates "frames" of two different types (MAnims and WAnims) - neither are stored in the executable.

Monsters can be attached to and removed from groups as well, and groups can be assigned their own AI types.

This engine is very dynamic compared to DOOM in these respects, and tons of stuff is data-driven. Unfortunately this also means tons of stuff is done through function pointers and the code is nigh impossible to follow in some places.

Also, the morphing is indeed dynamically generated, and it is hard-coded to work only on Veste's sprites :/ LoadMorph and the subroutines it calls are responsible for generating these morph graphics, and you could probably spend the better part of a lifetime understanding how this works precisely, without any source code to go on.

40oz: Yup. Dodging is strictly optional, although I hear it helps a lot ;)

EDIT: I have updated the map file at the same location.

[Gez]

Thanks. So, the monster behavior and stats are not in the exe itself, but actually in the maps themselves? For each map, there are the following files in SHADOW.LIB:
* mapname.arc - archetypes?
* mapname.crt - creatures?
* mapname.dor - doors
* mapname.itm - items
* mapname.map - map geometry
* mapname.ojt - ?
* mapname.scr - scripts
So I guess these data would be in .crt or maybe .arc.

The animations are stored in RAVDATA.DAT (floppy) or HD_CASTR.DAT (CD) in the animlist lump. (It has matching animliststart and animlistend marker lumps, so maybe it was even meant to be a cumulative system even if in the end they went with a single lump.) That lump defines a number of floor and wall animations. I guess that's what the WAnims code handles -- wall animations. And the MAnims code would be for monster animations.

[Super Jamie]

This is all way beyond my skill level, I just wanted to say WOW awesome work :)

[andrewj]

Gez said:

* mapname.ojt - ?

object?
objectives?

EDIT: looking at demo1.ojt, it appears to hold the scenery objects of the map.

EDIT2: the CRT files consist of records of length 202, with no header or trailer.

Here is a "map" of demo1.crt:

01 -- ca 01 7d 64 5a c8 0a 01 -- -- 1c 
01 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
01 ca ca 03 32 -- 64 19 0c 01 -- -- -- 
01 ca ca 04 fa -- 64 19 07 01 -- -- -- 
01 ca ca 05 04 -- -- 32 08 -- 08 ff -- 
02 ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
05 ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
06 ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
07 ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
08 ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
09 ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
0b ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
0c ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
0d ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
0e ca ca 01 7d 64 5a c8 0a 01 -- -- 1c 
02 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
03 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
04 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
05 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
06 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
07 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
08 ca ca 02 50 -- 50 28 0c 01 -- -- -- 
09 ca ca 02 50 -- 50 28 0a 01 -- -- -- 
0a ca ca 02 50 -- 50 28 0c 01 -- -- -- 
0b ca ca 02 3c -- 50 1e 0c 01 -- -- -- 
0c ca ca 02 3c -- 50 1e 0c 01 -- -- -- 
0d ca ca 02 3c -- 50 1e 0c 01 -- -- -- 
02 ca ca 04 fa -- 64 19 07 01 -- -- -- 
03 ca ca 04 fa -- 64 19 07 01 -- -- -- 
02 ca ca 03 46 -- 64 19 0c 01 -- -- -- 
03 ca ca 03 46 -- 64 19 0c 01 -- -- -- 
04 ca ca 03 46 -- 64 19 0c 01 -- -- -- 
05 ca ca 03 46 -- 64 19 0c 01 -- -- -- 
06 ca ca 03 46 -- 64 19 0c 01 -- -- -- 
07 ca ca 03 46 -- 64 19 0c 01 -- -- -- 
08 ca ca 03 1c -- 64 1a 0c 01 -- -- -- 
09 ca ca 03 46 -- 64 19 0c 01 -- -- -- 
14 ca -- 01 c8 -- 5a 2c 0a 01 -- -- 1c 

From that, I would deduce that the first and fourth longwords are the tile coordinates for creature. I'd guess that the fifth longword is the creature type/id.

Also the fifth creature stands out from the others (e.g. the missing 01), so perhaps that is the player spawn spot.