Shadowcaster modding 3: Disassembly

[Guest DILDOMASTER666]

I figured given that nothing has ever come of trying to mod Shadowcaster in the previous threads, I'll throw my hat into the ring again and revisit the idea from a different angle. In this thread, I'll document my attempts to disassemble RAVEN.EXE using IDA Pro, and will provide IDA databases for others to poke at as a starting point.

 

Of course, as is natural of a project of this type, function names and variable names are largely speculative except in instances where Raven left some handy error messages for me to trace back that included the name of the calling function.

 

In no particular order of importance, a few interesting finds to start things off:

 

- The game cannot touch more than 400 lumps at a time while running, enforced by loadstream() listed below. This could be a problem when adding many types of monsters and items to a custom map, for example.

- A more primitive form of Doom's zone allocation exists to accommodate the above, referred to as "disposable lumps" internally.

- There are some unused developer functions like dspstats(), which outputs your XP and current level, which could probably be triggered with a little memory editing at runtime.

- As expected, the .lib and .dat manipulation functions behave like precursors to Doom's wad functions. CA_GetNamedNum is an obvious standout, behaving almost identically to W_GetNumForName.

- Enemies can be assigned groups, which then can be assigned a number of different AIs. I have heard they can be assigned formations and will try to intelligently keep their relative positions to one another, but I have never seen this in-game nor found this in code.

- A sort of script language is implemented, based on a tile-event/condition system. You can think of it like primitive ACS that doesn't run in a VM and instead directly tells the exe to jump around apparently. It might have been a potential malware vector if abused sufficiently. I have not looked too deeply to see if any sanity checks are done.

- The map structure -- not just ceiling heights or textures or objects, but wall layout too -- can be manipulated at runtime. See PlaceTile().

- There seems to be a somewhat-arbitrary limit of 23 doors per level (?).

- DOS/32A will not work as a replacement for DOS/4GW with this game and I suspect it is because the exe is arranged in such a way that the game itself is considered an overlay. Is this normal?

 

The current version of my IDA database can be found here.

The current disassembly listing and C decompilation can be found here.

Edited July 7, 2017 by DILDOMASTER666

[Gez]

20 minutes ago, DILDOMASTER666 said:

- A sort of script language is implemented, based on a tile-event/condition system. You can think of it like primitive ACS that doesn't run in a VM and instead directly tells the exe to jump around apparently. It might have been a potential malware vector if abused sufficiently. I have not looked too deeply to see if any sanity checks are done.

That's interesting, and unfortunate. It's going to make any attempt at engine recreation that much harder.

[Guest DILDOMASTER666]

Just now, Gez said:

That's interesting, and unfortunate. It's going to make any attempt at engine recreation that much harder.

Absolutely, but not outside of the realm of possibility. It could in theory be implemented as a VM with additional sanity checks (assuming the ones in place, if any exist, are insufficient to prevent malicious use).

[Guest DILDOMASTER666]

Sorry for the double post. I updated the IDA database with a bunch of things ranging from sound code and a bunch of init functions for various startup elements to preliminary work on deciphering the scripts subsystem.

 

Things marked with SCR_* belong to the scripts subsystem. Variables prefixed with "qq" mean I am not certain that's what the variable is actually for, but I have done my best to make an initial educated guess. Functions prefixed with an underscore indicate the same thing.

 

I have also exported a disassembly listing so that people who don't have IDA but still want to follow a long can have a look. That can be found here or in the OP.

[Guest DILDOMASTER666]

Triple post, great.

 

I updated the IDA database, listfile, and added a decompiled C listing (see RAVEN_list.zip).

 

A lot of script functions were explored. A lot of them are reasonably sane and only one or two that I have seen actually tell the program to jump around in an unsafe way. More sound code was poked at, and there are now a bunch of early disassemblies of map functions including LoadMap(), which should prove helpful in making a map editor at some point.

 

There is some kind of subsystem related to "tile content" that I do not yet understand, which is apparently different from tile events, scripts, items, et cetera.

[Guest DILDOMASTER666]

Intermediary update.

 

I performed a cursory examination of many functions by tracing back calls to SC_Error(), a conjectural name given to a function called in many places to throw errors. Among interesting finds were an exact format for several map data structures, some of it known, some not. There is not yet enough known to make a complete map editor, namely the format of cellobjects is not completely known and there is still next to no documentation on scriptfuncs/scriptconditions.

 

I will update later today with a basic map spec and an updated IDA database, C and assembly listings containing the new findings, including at least some examination, naming and type setup for every function that calls SC_Error(). Some good stuff in there like MaskDrawPic, DrawSubPic, MaskDrawSubPic, LoadGame, SaveGame, some functions related to Veste's morphing, HUD stuff, and more.

 

Around 300 of the 1750 or so functions/subs in the executable will be touched in the update.

Edited July 8, 2017 by DILDOMASTER666

[VGA]

I love reverse engineering efforts :D

[Guest DILDOMASTER666]

Database, decompiled C and ASM listing have been updated. Notable changes include some gameplay functions, lots of renderer functions, an early look at a bunch of stuff like LoadGame/SaveGame/LoadMap, and some deeper attempts at documenting some of the script functions.

 

As promised, the map format:

 

//The order in which things are loaded and thus the arrangement of data is like so:
{
  maplump = CA_IndexLump(mapname, &shadowlibptr);
  fread(tilemapfloors, 2048, 1, shadowlibstream);
  fread(floorheight, 1024, 1, shadowlibstream);
  fread(tilemapceilings, 2048, 1, shadowlibstream);
  fread(ceilheight, 1024, 1, shadowlibstream);
  fread(tilemapwallsnorth, 2048, 1, shadowlibstream);
  fread(unknownwallnorthdata, 1024, 1, shadowlibstream);
  fread(tilemapwallswest, 2048, 1, shadowlibstream);
  fread(unknownwallwestdata, 1024, 1, shadowlibstream);
  fread(packeddoors, 1024, 1, shadowlibstream);
  fread(&unusedmapdata, 1024, 1, shadowlibstream);
  for ( i = 0; i < 1024; ++i )
    *&tilemapunpackeddoors[2 * i] = packeddoors[i];
}
/*Thus, this can be abstracted like so:

  floor       = unsigned short[1024]
  floorheight = signed char[1024]
  ceiling     = unsigned short[1024]
  ceilheight  = signed char[1024]
  wallsnorth  = unsigned short[1024]
  unk_walln   = signed char[1024]
  wallswest   = unsigned short[1024]
  unk_wallw   = signed char[1024]
  door        = signed char[1024]
  unused      = (char)[1024]

Doors are then "unpacked" into (probably) unsigned shorts in memory.*/

 

Note that while doors are indeed defined in some way in *.dor files, their placement on the level is strictly contained in the .map. Notably missing are cellobjects and tileevents. Cellobjects are almost 100% certainly in *.ojt files, whereas data related to tileevents could be in unk_walln/unk_wallw and are almost certainly in *.scr files. In addition, while memory is allocated and used for determining the height of a floor cell, no mechanism exists by which to make use of this data without visual glitches. That is to say you can set the floor height of a tile to whatever you want, but the engine has serious issues when you have areas of differing height like this. The feature was obviously not finished.

 

Ceiling height changes are another story. You can have areas of different ceiling heights, but only in specific circumstances and some additional data is required somewhere to make "slopes". unk_walln/unk_wallw is possibly used here too, to define in what direction and height the renderer should draw the top edge of the texture.

 

There may be a total of 23 doors placed in a *.map. There may be a total of 84 cellobjects placed. Exceeding these limits will crash the game because the arrays that hold the data for these structures is static in size.

 

Tilecontent structures appear to indicate the presence of any of these elements in a cell, but the array that is written to to indicate the presence of tilecontent appears to hold significant values which I do not understand yet.

Edited July 8, 2017 by DILDOMASTER666

[Arno]

On ‎5‎-‎7‎-‎2017 at 11:33 AM, DILDOMASTER666 said:

There seems to be a somewhat-arbitrary limit of 23 doors per level (?).

The programmer of the game could probably have easily reserved more space for doors. But apparently this limit of 23 doors fitted the needs of the level designers at that time. By comparison, Wolfenstein 3D has a maximum of 64 doors per level. The Catacomb Abyss has the maximum set to 5 doors.

Edited July 11, 2017 by Arno

[Gez]

Very interesting stuff; keep it up!

[Linguica]

I agree, keep up the good work, Dildo Master 666.

[R1ck]

Following. Might learned a thing or two. 

[Guest DILDOMASTER666]

Fruits of labor since last update:

 

(bonus: mod loading support)

[Arno]

Very nice! Apparently you decided to take your documentation efforts one step further, to what seems to be a first working build of an engine recreation.

So are there any specific features you are planning to work on next?

[Guest DILDOMASTER666]

Yes. As soon as possible, I want to implement the following:

-externalize certain game elements to scripts, such as the quest progress text, level progression, and morph properties
-in-engine level and archive editor
-a few specific bugfixes, like the crash on lavamines.map in the original exe if you try to pick up the boulders after lava has flowed into the ice room

In addition, the current implementation of a lot of things is really messy. An example, texture loading happens through a particularly heinous function called R_ConstructWallPicDebug() which creates GPU surfaces and loops over 64*height data, converting the raw data pointed to, skipping the column directory. This is obviously incorrect and will be thrown out and rewritten before a first release, but it was enough to "just make it run" for now.

 

The engine is written within GameMaker: Studio. Although I anticipate that you will be able to modify almost all of the game without ever touching the project sourcefile, the version used to build each engine executable will be freely available. Despite the working title, I would like to continue thinking positively about Mac and Linux support, which should be a somewhat straightforward endeavor with GMS.

 

EDIT: And despite the implications of using GameMaker: Studio, I assure you that no data from the game will be statically linked with the executable. To be clear, this is not a remake using converted original assets. This is still pulling all its data from a legitimate game installation; it's just written targeting Game Maker so my awful code will always compile for everybody.

 

 

Edited February 13, 2018 by DILDOMASTER666

[Guest DILDOMASTER666]

Find attached the first (and most primitive) form of the "source port".

 

It will not run unless you supply the original game data. Both floppy versions 1.12F and the CD version are supported. See the included readme.txt for where to place your game data; it does not go in the program directory.

WindowCaster-v1.zip

 

This version only runs one level: ruinsb. Sprite collision detection does not exist, there is no way to change levels, and there's a bug where opening the quit menu while a game surface is being displayed on the main menu will cause the game surface to overlap it. Arrows move.

[Guest DILDOMASTER666]

https://www.dropbox.com/s/afr8zqam2mtfil1/WindowCaster_dev_5a85151f.zip?dl=0

 

Find linked the first "real" version of WindowCaster. It is now an actual functional Shadowcaster map viewer, although most of the cell contentids are still unmapped so only ruinsb is accessible. A lot of code was completely rewritten from scratch.

 

From changelog:

Quote

ver. 5a85151f:
  * completely rewrote the collision detection from scratch. it is now objectively 200,000 times better than the old code. sprite collision works correctly and uses most of the same code, but level objects are not loaded yet so you cannot test this.
  + preliminary audio support. not finished and not accessible now.
  + maps can be loaded via CA_ConstructMapDebug(), if you know the offset to the map file in the lib. CA_CheckNumForName() does not work right at the moment, so this is the only method.
  + morphing works, but not all forms are accessible right now. you can currently change to human, Maorin, Caun, Opsis, and Kahpa.
  + for now, all graphics are statically linked with the executable. this is because R_ConstructWallPicDebug() is *very* glitchy. do not depend on this function, please.
  - got rid of the old map data that was statically-linked in the exe. as above, the game now reads maps directly from the libs and creates cell content instances as necessary.

 

Known bugs:

Quote

  * due to drastic changes in the way the game rendersurface is cached, the gui on the main menu is broken for now. your clicks will still register, but you cannot see the cursor. investigating a fix.
  * ruinbwat is broken. yes, i know. that level doesn't load properly with CA_ConstructMapDebug() just yet, due to missing mappings to cell content ids. you also cannot surface from this level yet. whether or not the player can surface from underwater depends on the contentid of the ceiling tile you are under, and since no tiles get instanced for the ceiling yet, you are stuck if you submerge to ruinbwat. restart WindowCaster if you really want to continue playing on ruinsb.

 

EDIT: morphing is done with the number keys, 1-5. forgot to mention that in the readme.

[VGA]

Has anyone tried it? I don't have the files, maybe I will acquire them later *cough*

[Guest DILDOMASTER666]

Please go here for a more up to date version of WindowCaster, a lot has been improved upon since the release above