ReflectionHLE (Reflection Keen)

[NY00123]

[UPDATE: Oct 14 2022] Web page link for reading a bit about the project: https://reflectionhle.com/

 

[UPDATE: Dec 31 2021] The project is titled ReflectionHLE as of writing this. Ignoring the URL, the rest of this post currently remains unedited by me.

 

Hi all,

Let me introduce you to the following source ports inspired by Chocolate Doom, now also covering Wolfenstein 3D: https://github.com/ReflectionHLE/ReflectionHLE/releases

This is a 6 years anniversary release. On September 26 2014, I released a new port of Keen Dreams, initially titled "Chocolate Keen Dreams".
Although Keen Dreams was released earlier in the same month, I actually started some work on it even earlier, after the Catacombs were open-sourced (in June 2014). Reason is that I already knew that a lot of the code was common.

Following Keen Dreams, I added Catacomb Abyss, then Catacomb 3-D, and finally the rest of the Catacomb Adventure Series.

Recently, I've been working on the source ports again. With the assistance of the gamesrc-ver-recreation/wolf3d tree, the last release introduces support for Wolfenstein 3D, Spear of Destiny (excluding the mission packs) and Super 3-D Noah's Ark (DOS version).

For Wolfenstein 3D, this currently covers Apogee shareware versions 1.0, 1.1, 1.2 and 1.4 (the one with cheats), as well as 6-episodes Activision v1.4.
For Spear of Destiny, this covers FormGen demo v1.0 and Activision v1.4.
For Super 3-D Noah's Ark, this covers the one Wisdom Tree DOS version that I'm aware of.

There are still some problems. In particular:
- You can't save games or load saved games in Wolf3D and the games based on it.
- These games have no support for the "modern" game controller scheme. I think that the other games should still have it, though.
- For technical reasons, you can't load the Spear of Destiny mission packs.
- More generally, you can't load TCs made to work with original EXEs from the 90s (as released by id and/or related publishers). I still haven't decided how to approach this.
- In order to detect game data, you also need the original DOS exe (albeit the 2015 edition of Keen Dreams is an exception).
- For Super 3-D Noah's Ark, this exe currently has to be named noah3dos.exe, as this is the way it arrives from Steam right now. Currently, there's no good way to support auto-detection of data with alternative file names.
- The rate of palette updates in Wolfenstein 3D is at least somewhat imprecise.
- VSync is disabled by default for now. There are other potential problems with timing in Wolfenstein 3D, which might partially be related to instances in which the game tries to render more often than the host display's refresh rate.
- Stereo panning remains unimplemented.
- For the Wolfenstein 3D based games, if you try to use a wall right after pushing it, the behaviors are essentially undefined.

Edited October 14, 2022 by NY00123 : New project name

[xvertigox]

That's an interesting jump from Keen to Wolf support. I was looking for Blake Stone port just yesterday actually but bstone has no automap feature zz.

 

Good work, dude.

[Arno]

Congratulations @NY00123, the addition of Wolfenstein 3D support is quite an impressive feature!

 

I've tried the win64 version on Windows 10 and noticed that I could not browse my c: drive. It shows up as having no folders. Browsing my home dir did work, so I just copied the game files there.

 

I just played through the first level of Catacomb Abyss, as well as E1L1 of Wolfenstein 3D without any problems. Looking good so far!

[NY00123]

Thanks for the feedbacks so far!

 

2 hours ago, Arno said:

I've tried the win64 version on Windows 10 and noticed that I could not browse my c: drive. It shows up as having no folders. Browsing my home dir did work, so I just copied the game files there.

 

Yeah, I got a similar report from someone else about this. It's basically another example of a bug I wasn't aware of, due to the project not being as commonly used, and/or due to not having a good place for bug reports. Basically, the program was inspecting the contents of "C:" instead of "C:\". Here, my understanding is that "C:" refers to the most recently set directory in the C drive.

 

I've actually just uploaded a hotfix release which should resolve this problem, as well as a stale menu item non-deterministically being reproduced in SOD and the Activision version of Wolf3D, depending on the way the exe is built. The usual "Releases" page on GitHub should have the last release.

 

Sep 28, 2020 (v0.30.1):

* Don't show controller settings specific to the Catacombs for Wolf3D.
* When letting the user choose a root drive on Windows for locating
game data, append a backslash. For instance, use "C:\" instead of "C:".
Otherwise, the wrong locations may get accessed with _topendir.
* Fix dir resource leak in BEL_Cross_DirSelection_PrepareDirsAndGetNames.
* Wolf3D: Fix the occurrence of a stale menu item in the
main menu for the GOODTIMES and SPEAR versions.
* Improve the scrolling in the launcher a little bit,
mostly while pressing on up/down.

 

[ETTiNGRiNDER]

20 hours ago, xvertigox said:

That's an interesting jump from Keen to Wolf support. I was looking for Blake Stone port just yesterday actually but bstone has no automap feature zz.

 

Good work, dude.

That would be a bizarre regression, since vanilla Blake Stone had an automap to begin with?

[xvertigox]

3 hours ago, ETTiNGRiNDER said:

That would be a bizarre regression, since vanilla Blake Stone had an automap to begin with?

 

Spoiler

 

I haven't played in so long that I assumed that Wolf engine = no automap but in thinking about it rott had one too right? Guess it's time to fire up some Blake Stone.

[Arno]

22 hours ago, NY00123 said:

Yeah, I got a similar report from someone else about this. It's basically another example of a bug I wasn't aware of, due to the project not being as commonly used, and/or due to not having a good place for bug reports.

 

Thanks for the hot-fix! Browsing the C: drive works fine now.

IMO, given the amount of games and game versions supported, combined with the various platforms that are supported, I think your source port works remarkably well.

 

I've seen quite a lot of the Catacomb games myself, so I might be able to help you with some testing. Below are two bugs I encountered. 

 

I have the GOG Catacombs Pack installed, but it's not automatically detected. That's because at one point in time GOG.com updated the installation package and changed the location in the registry, which means now two registry locations will have to be checked. In be_cross_fs.c you already check for "SOFTWARE\\GOG.COM\\GOGCATACOMBSPACK". The new location is at "SOFTWARE\\GOG.com\\GAMES\\1207659189". The number apparently represents the product ID: https://www.gogdb.org/product/1207659189 . For the same reason my GOG Wolfenstein 3D installation isn't detected either. That one is located at "SOFTWARE\\GOG.com\\GAMES\\1441705046", see https://www.gogdb.org/product/1441705046 .

 

In Catacomb Abyss I get a crash when trying to travel to map slot 19 with the F10+W cheat. Although the dialog box asks for a number 0-18, it is actually possible to go to map 19 when playing Catacomb Abyss in DosBox. See https://catacomb.miraheze.org/wiki/Tile_Tester_3 . 

 

[NY00123]

On 9/29/2020 at 12:18 AM, Arno said:

 

Thanks for the hot-fix! Browsing the C: drive works fine now.

IMO, given the amount of games and game versions supported, combined with the various platforms that are supported, I think your source port works remarkably well.

 

I've seen quite a lot of the Catacomb games myself, so I might be able to help you with some testing. Below are two bugs I encountered.

 

Thanks for the additional feedback and testing, and further for confirming that you can browse the C: drive now.

 

On 9/29/2020 at 12:18 AM, Arno said:

 

I have the GOG Catacombs Pack installed, but it's not automatically detected. That's because at one point in time GOG.com updated the installation package and changed the location in the registry, which means now two registry locations will have to be checked. In be_cross_fs.c you already check for "SOFTWARE\\GOG.COM\\GOGCATACOMBSPACK". The new location is at "SOFTWARE\\GOG.com\\GAMES\\1207659189". The number apparently represents the product ID: https://www.gogdb.org/product/1207659189 . For the same reason my GOG Wolfenstein 3D installation isn't detected either. That one is located at "SOFTWARE\\GOG.com\\GAMES\\1441705046", see https://www.gogdb.org/product/1441705046 .

 

 

Yeah, I also spotted this on my own, after re-checking with an updated installer from GOG.com. I still haven't added detection of GOG.com installations of Wolfenstein 3D and other games based on it.

 

On 9/29/2020 at 12:18 AM, Arno said:

 

 

In Catacomb Abyss I get a crash when trying to travel to map slot 19 with the F10+W cheat. Although the dialog box asks for a number 0-18, it is actually possible to go to map 19 when playing Catacomb Abyss in DosBox. See https://catacomb.miraheze.org/wiki/Tile_Tester_3 . 

 

 

I can reproduce the problem. At least in this case, there's a map that can be loaded, but I think that with later, non-existent maps, the behaviors are more-or-less undefined. Keen Dreams v1.00 has a similar issue with empty map slots.

In Keen Dreams v1.00, ((mapfiletype     id0_seg *)tinf)->headersize[mapnum] was set to 0 for mapnum == 6, meaning that I can detect this case in CA_CacheMap and then possibly report a new (non-vanilla) error. With the original DOS EXE, you might get one possible error out of two, which isn't related to the lack of a map, and/or possibly also a hang. Thus, there's probably no problem with making the port quit with a specific error message in this case.

 

For the Catacombs, this should still be checked, since CA_CacheMap is a bit different, but it's probably similar.

 

Let's get back to Catacomb Abyss's map slot 19, though. Basically, the problem here is that the levelnames array is defined to have (pointers to) 19 string literals, so it's an instance of overflow.

 

I checked the layout of the original EXE from v1.13 (in unpacked form). Hopefully, I have no typo in the following contents.

 

Here, the levelnames array, located at dseg:1300, is immediately followed by the level name strings themselves. Thus, levelnames[19] technically translates to the first two bytes of the string "The Towne Cemetery", which are 0x6854 (in Little-Endian order).

 

This implies that it the code in question tries to read a string literal from the grsegs array. Why? Well, NUMCHUNKS is 626, implying that the grsegs array has the size of 1252 i.e., 0x4E4 bytes. This is exactly the difference between the offsets of the following addresses in the EXE:

208F:65BE       _grsegs
208F:6AA2       _starting_level

Now, since levelnames[19] is 0x6854, this translates to chunk no. (0x6854-0x65BE)/2  == 0x14B i.e., 331. Thus, it is a 16x16 background tile. However, my understanding is that such tiles never get cached, because the code used for showing the overhead map is disabled. Therefore, in practice, grsegs[331] == 0, so when levelnames[19] is accessed, you get an empty string.

 

[NY00123]

ok, I've tried to investigate the specific case of Catacomb Abyss level 19, and it's a bit more tricky.

 

First of all, I should say that what I've written earlier applies to versions 1.13 and 1.24 altogether. Main difference is that for v1.24, grsegs resides at address 2082:657E. Since levelnames[19] is still 0x6854 (i.e., the first 2 bytes of "The Towne Cemetery"), this translates to chunk no. (0x6854-0x657E)/2 == 0x16B i.e., 363. This is again a 16x16 background tile, and the explanation from my preceding post still applies.

 

So, let's assume that we do process levelnames[19] as if it were an empty string. What's to follow?

Well, we reach CA_CacheMap, which attempts to access mapheaderseg[19]. This leads to another buffer overflow.

For the EXEs from versions 1.13 and 1.24, mapheaderseg[19] basically overlaps textstarts[0].

 

Generally, textstarts[0] should be set to 0 (on exe load and via ScanText).

If we enter level 19 from another level, then CA_CacheMap will simply use mapheaderseg[19] i.e., textstarts[0] to store a pointer to newly allocated data.

SetupGameLevel will later reference the map width and height from this pointer.

 

Later, ScanText will be called from CacheScaleds, thus resetting textstarts[0] i.e., mapheaderseg[19] to 0.

What this effectively leads to is a memory leak.

 

When leaving level 19 (or possibly even just restarting level 19), CA_CacheMap will not free the memory previously referenced by mapheaderseg[19], because this pointer variable, which is actually textstarts[0], has already been set to 0.

 

One possible way to handle this in a port is as follows. The mapheaderseg array length can be increased by 1, and then mapheaderseg[19] (i.e., mapheaderseg[NUMMAPS+1]) can be reset to 0 right after being used in SetupGameLevel.

 

I'm unsure about any later level. I'll first ignore possible issues with levelnames. Just like mapheaderseg[19] refers to textstarts[0], mapheaderseg[20], mapheaderseg[21] and mapheaderseg[22] refer to the following 3 locations in the textstarts array. Main difference is that these offsets are generally nonzero when CA_CacheMap is called. Thus, it'll try to set their purge levels with MM_SetPurge.

 

Because these aren't properly allocated memory addresses (at least not in my tests), you'll simply end with this error message: "MM_SetPurge: Block not found!"

 

An exception to the above is level 21 in v1.13. In that case, I'm not currently sure what exactly occurs, but it looks like a case of buffer overflow. I get a minor screen corruption, followed by a crash of DOSBox.

 

If it isn't a case of levelnames[21] (which seems to reference one of the values in shapesize) leading to an overflow in a call to US_CPrint, then maybe MM_SetPurge actually finds a matching address, and thus we get some problems.

 

LATE UPDATE: ok, I think that I've found the cause of the problem. While there's no overflow in the length of levelnames[21], this is how the string looks like as a sequence of bytes: 2F 06 15 07 73 33 00

 

What happens is that ShiftPropChar from ID_VW_AE.ASM should be called once per char. When it's called with the value of bx == 6, [es:charwidth+bx] (i.e., currfont->width[charnum] in Reflection Catacomb) is 0, so the si register is also set to 0. It remains set to 0 even after a few more manipulations on it. Eventually, an attempt to jump is done as follows:

 

jmp     [ss:shiftdrawtable+si]

However, because si set to 0, and shiftdrawtable further begins with 0, this effectively jumps to some arbitrary address.

In that case, it turns out to be exactly the location where VW_Vlin is called from VWB_Vlin, which must be the explanation for the vertical lines shown in the screenshot.

Of course, the return from VWB_Vlin leads to a more-or-less undefined location.

 

 

Edited October 1, 2020 by NY00123

[Arno]

I sure wasn't expecting such a detailed analysis when I briefly mentioned level19. Interesting!

When I learned about the possibility to warp beyond level 18, combined with the fact that GAMEMAPS.ABS actually contains 23 levels, my first thought was how to get all the hidden levels to load. Because I wanted to see in first person view what kind of levels the developers made for test purposes.

It's nice to see how you approach the very same topic from a different viewpoint. I now fully realise that the "chocolate" philosophy brings its own challenges. It seems certainly not trivial to accurately replicate bugs, when the original memory layout is lost due to the usage of a modern compiler.

[Blzut3]

2 hours ago, Arno said:

when the original memory layout is lost due to the usage of a modern compiler.

Although sounds like not quite so much in this particular case it seems, it's not just using a modern compiler but also the CPU architecture change.  x86-16 had interesting memory wrapping properties due to the way memory was addressed.  Plus primitive types changing sizes (especially pointers since integer sizes can usually be forced) all means even if things were laid out in the same order things still wouldn't line up correctly once built as x86-32 or x86-64.

[NY00123]

I'm actually unsure that I got to visit all warp-only levels in the Catacombs, not to mention the generally inaccessible levels of The Catacomb Abyss. If it weren't for checking in an editor or another source port, I'd probably assume that these were not that far from empty map slots.

 

The comment about memory wrapping is true. It was especially important while implementing the CGA graphics routines for Keen Dreams, which generally output the graphics to a buffer in system memory instead of video memory. I remember running into problems with this, until I read a comment about memory wrapping from David Gow (developer of Omnispeak and the SDL2 port of Keen Dreams), which was actually related to EGA graphics.

 

For the EGA graphics, Reflection Keen is doing things a bit differently; Certain aspects of the EGA/VGA are emulated, at least up to some limited levels. So I was already forced to work with addresses that wrap, either way.

 

I don't know how many behaviors that depend on buffer overflows are present in Vanilla Doom and modifications made for it, and how much can Chocolate Doom be impacted. For a couple of Doom examples, I'm aware of the tutti-frutti and Medusa effects. I think that Medusa can be quite difficult to emulate, if it's feasible at all.

 

For Wolf3D, my current impression (no idea if it's true) is that there are more ways in which buffer overflows can impact the game, especially in add-ons.

 

As briefly hinted in my initial post, one example I've learnt about only recently, impacting even the original games: If you press on a pushwall right after pushing it, then depending on the wall tile, you might open/close a door with an arbitrary door number. If this number is >= MAXDOORS, you get an overflow. One possible side-effect is a change to some tile in the map, say by making it blocking.

Edited October 4, 2020 by NY00123

[NY00123]

A new release is now available. The most important update is probably the addition of support for the Spear of Destiny Mission Packs. There's more, like the detection of more possible locations for game installations, as well as support for more versions of Wolfenstein 3D.

 

Oct 16, 2020 (v0.31.0):

* Support more than one possible filename for game data and EXE names.
Following this, a file named "NOAH3D.EXE" can now be identified as
a part of Super 3-D Noah's Ark, in addition to "NOAH3DOS.EXE".
This refers just to the matching DOS executable.
File integrity check is done using the size and crc32.
* Additionally, again thanks to the above, support for the
Spear of Destiny Mission Packs is now in. Reason it's related
is that the file extension of the mission-specific game data can vary,
depending on the game installation being in use and its current state.
For all missions (including the original), .SOD and .SD4 are valid
file extensions. Other allowed extensions: .SD1 for the base game,
.SD2 for mission 2 and .SD3 for mission 3.
* Note that as a side-effect, data integrity checks are now done for all
game data (graphics, maps, sounds) read from the Wolfenstein 3D codebase.
In addition to earlier uses of it, the hidden "manualgamevergame" setting
can now be used to disable this, albeit this might make it impossible
to properly start Super 3-D Noah's Ark or any of the Spear of Destiny
missions, depending on the filenames.
* Add support for more versions of Wolfenstein 3D, again by using code from
gamesrc-ver-recreation. This covers Apogee 6-episodes versions 1.1, 1.2 and
1.4, two GT Interactive releases, an id Software release, Apogee 3-episodes
versions 1.2 and 1.4, an Imagineer version and the March '92 prototype.
(Thanks to the wolf3denginepatches repository and Chris for assistance.)
* Add detection of Wolfenstein 3D as it used to be offered via the
last incarnation of 3drealms.com, either on its own or as a part
of the 3D Realms Anthology. (Thanks Lunick.)
* Add detection of Wolfenstein 3D, Spear of Destiny and Super 3-D Noah's Ark
as currently available from GOG.com. (Thanks WayneCa and chungy.)
* Improve detection of game installations which is done by inspecting
the Windows Registry. Basically, this is now done by attempting to
inspect data for 32-bit and 64-bit applications altogether, while
additionally still trying to get data with SHGetValue as usual.
* Wolf3D: The UPLOAD macro was replaced with an UPLOAD variable.
* Internally rename the wl1ap12 namespace to wl6ap11, and do the same with
other mentions of such strings. Most noticeable impact is probably
that when separate Wolf3D EXEs are build, reflection-wl6ap11 is
created instead of reflection-wl6ap12.
* Typo fix: Do show "Wolfenstein 3D Shareware v1.4 (Local)" when referring
to shareware v1.4, instead of mistakenly mentioning v1.2 in the string.
* Use a bit shorter descriptions for certain Wolfenstein 3D and
Spear of Destiny versions, so they can fit as launcher titles
(BE_Launcher_Handler_SupportedGameVersionSelection).
* Add high level XMS and EMS implementations, use them in Wolf3D's
Page Manager, and also remove the REFKEEN_SIMPLIFIED/NOEMS/NOXMS
implementations from Wolf3D. (Thanks Blzut3.)
* Wolf3D: Following the removal of the REFKEEN_SIMPLIFIED-specific code
from id_pm.c, pointers can now be appropriately converted in ShapeTest.
* A small fix for using page up/down in the launcher while
none of the menu items can be selected.
* A couple of fixes for bugs introduced after the audio mixer refactoring,
specific to runs without the audio subsystem.
* Separate more audio and timing related code into more files, while reducing
direct references to SDL (albeit it's still used in the same manner).
* Generally, continue splitting the contents of be_cross_fs.c and
making other restructurings and differing changes related to this code.
* Wolf3D: Fix a possible buffer overflow in US_CheckParm, reproduced by
passing an argument with no alphabetical character. A good example is
the second argument in the following arguments string: "-tedlevel 1".
* Make the codebase more usable with Visual Studio.
* Launcher: Add a missing comment about the impact of the
scale type setting on the border porches.
* Bug fix: Conditionally hide or show the mouse cursor after
toggling fullscreen, depending on game-specific setups.
* Fix an instance in which the in-game contents don't get
appropriately resized after toggling fullscreen.
* Wolf3D: Make the mouse cursor usable while the game
is paused, if a fullscreen isn't used.
* Other misc. fixes/changes.

 

Edited October 16, 2020 by NY00123

[NY00123]

I'm ready to announce another release. It introduces saved games support to Wolf3D. They should be written in the same binary formats as the matching DOS versions. On the other hand, due to the nature of the implementation, there may still be bugs.

I also fixed a bug which, unfortunately, may break (partial) compatibility with Keen Dreams and Catacombs saved games. Then again, such compatibility was probably already broken in the same manner with the matching DOS versions.

 

Dec 25, 2020 (v0.32.0):

* KDreams, Catacombs (and Wolf3D): Make it possible
to bind actions to a game controller's D-pad.
* Wolf3D: Support game controllers, in a similar manner to
what's already been covered for Keen Dreams and the Catacombs.
Further add mappings for touch input.
* Wolf3D: Support stereo panning. This might not be fully accurate,
but hopefully, it's not that far.
* Wolf3D: Support saved games. They should be read and written using the same
formats as the original DOS versions. This may currently be buggy by nature.
* SD_TimeCountWaitFromSrc/SD_TimeCountWaitForDest timing fix, applying to
all games, but especially noticeable while playing back demos in Wolf3D.
* Add support for Spear of Destiny FormGen version 1.0, as well as the
variation of FormGen version 1.4 which is actually identified as "V1.4"
in-game. Currently, the one mistakenly identified as "V1.0" won't be detected.
The mission packs' data still needs to be the same as in the
Activision version, thus e.g., including the UAC logs.
* When game installations are shown in the launcher, don't print their
locations for now. This might become more useful later, in case it will
be possible to choose any of multiple game installations which otherwise
match the same version (e.g., due to somewhat differing game data).
It shouldn't matter as much for now.
* A change that may potentially break saved games made
with older versions of the KDreams and Catacombs ports:
The macros COMPAT_OBJ_CONVERT_OBJ_PTR_TO_DOS_PTR and
COMPAT_OBJ_CONVERT_DOS_PTR_TO_OBJ_PTR were fixed. The original sizes
of the object structs, as present in the DOS executables from the 90s,
are now used, instead of sizeof(objtype) from the source ports.
* Other misc. fixes and modifications. Thanks to Blzut3 for
assistance with a subset of the changes in this version.

 

[Arno]

Well done, @NY00123!

I was waiting for the save/load functionality, in order to properly revisit Wolf3D. With the latest version, auto detection of my GOG (Activision) installation worked right away.
In the early 90's I used to play Wolf3D keyboard-only with PC speaker sounds. At first I did the same in Reflection Wolf3D. I gotta say, that does give some nice nostalgic vibes, especially the treasure pickup sounds. Nevertheless, most of the levels I played with the comfort of digitized sounds and keyboard/mouse. I know that the original Wolf3D did not support WASD, but fortunately I was able to configure the controls in Reflection Wolf3D in a way that still worked well for me. I did turning, shooting and opening doors with the mouse, while the W/S keys were configured to forward/backward.
I played through the entire first episode, secret level included, and did not encounter any bugs. It was great fun. Good job!

[NY00123]

Thanks @Arno! Great to see that this brought you back to playing Wolf3D's first episode, initially even doing so the way you originally did in the early 90s.

It's also good to see that you didn't encounter a bug. There's still the aforementioned vanilla bug related to pushwalls, not faithfully replicated at the moment.

Vanilla Wolf3D indeed doesn't support concurrent use of a WASD keyboard layout and the mouse for turning. Something like DOSBox' mapper can let you assign separate keys for strafing, but Vanilla Wolf3D still doesn't let you turn and strafe concurrently.

 

This might be a good chance to see if anybody has a comment on a bit technical topic. I can't say if I'll change anything, but right now, the way the ports handle user input has limitations.

- Since the ports aim to be Chocolate Doom equivalents, it should be possible to use a keyboard and mouse in ways which behave more-or-less the same as under DOS.

- For instance, when Wolf3D's IN_ClearKeysDown function is called, it should have the same effects as in the original exes.

- Also, unlike Doom, the games Keen Dreams, Catacomb 3-D and Wolfenstein 3D let you select keyboard and mouse actions from the game itself.

 

Question is how to make this work when you want to use a different input device, not originally supported in the 90s (at least not in the same manner). What I did for now was the following:

- Some form of support was added for modern game controllers and touchscreens. When in use, what the game sees in practice is still the old-style keyboard and mouse events.

- This works by translating uses of the input devices via internal mapping tables. The tables change according to context (e.g., menu navigation vs. general gameplay).

 

Unfortunately, this approach has obvious disadvantages. Just for a few examples:

- You cannot configure the keyboard in ways which the game didn't originally allow you to. In particular, you can't configure it at all for the Catacomb Adventure Series.

- If you assigned the same key to two actions in the game's menus, each of the controller buttons mapped to one of these actions might also, in practice, trigger both actions.

 

While originally adding the support to Keen Dreams, initially, I considered disabling the keyboard settings while a modern game controller is in use. However, I also wanted such controllers to be usable out of the box via hot-plugging, at least by default. Therefore, this eventually didn't look like an option to me.

 

To finish, if you check any of the "altcontroller.c" files, currently present in the source ports and covering input mappings, you can see that - well - they aren't exactly simple txt or xml files which you may write in a short time. Obviously, describing locations of touch input controls without visual input is expected to be more difficult, but even if we ignore touch input, properly creating the mappings needs some work.

[Dweller Dark]

As far as I can tell, Reflection works great for Wolf3D and Spear of Destiny for me. I just found out about this thread and now I can play Wolf3D/SoD, so this is awesome. I appreciate the work you put into making this!

[Taw Tu'lki]

That's interesting.

For me as a fan of Wolf 3D and Keen I should try it.

[NY00123]

Thanks for the interest in the ports!

 

I've had a bit of chat with Blzut3 on my preceding post, emphasis on support for input devices.

It's still early to say if anything will be changed, but I'll at least bring a possible suggestion.

 

Right now, you can use the launcher to bind game controller buttons to in-game actions, like jumping in Keen Dreams.

In practice, these buttons are actually mapped to the corresponding keys (as originally used under DOS). For jumping in Keen Dreams, this is "Ctrl" by default.

If you change this key in Keen Dreams' control panel, the game controller mapping will be updated accordingly.

 

As a side-effect, if you start Keen Dreams, change the jump key to "F1", then start a game and press on a game controller button mapped to jumping, chances are you'll see the help sections instead.

 

Thus, the suggested change was to simply map game controller buttons to very specific keys. For instance, instead of mapping a button to "Jump", it'll be mapped to "Ctrl". Of course, this means that the button will not be usable for jumping if you change the jump key the way it's originally done under DOS.

 

I'm not entirely sure which option (currently used or suggested one) is better, or at least less confusing for the user, but the suggested one will be simpler to implement.

 

One may similarly consider launcher-side keyboard overrides, so you can use e.g., keys that the game wouldn't even let you choose at all. There should probably be no such overrides at all by default.

 

 

Edited March 4, 2021 by NY00123

[Arno]

I've given your dilemma on the support for input devices some thought, but I suspect there isn't a perfect solution.

 

In other games where keyboard, mouse and game controller all need to work side-by-side, it is often allowed to bind multiple keys/buttons to the same in-game action. I can imagine that in case of the Catacomb Adventure series, such functionality could work well from the launcher. These games already bind (hard-coded) multiple keys to the same action. For example, the launcher could show by default that "C" and "SPACE" are bound to the cure potion, but with the option to replace those by different keys or game controller buttons.

 

Like you said, it gets complicated in case of Keen Dreams, Catacomb 3D and Wolfenstein 3D, where there is an in-game control panel that lets you rebind only a single key per action. And not necessarily every key that might exist on a modern keyboard. What could be an idea is to have an in-game control panel that works authentically with single keys, combined with a launcher-side control panel that shows the in-game bindings as read-only, but with the ability to bind additional keys.

 

 

[NY00123]

So, a maintenance release is now available. It should fix a few issues related to the use of game controllers, including a possible crash. If you're using SDL 2.0.14, you can also use the Xbox Elite controller's paddles.

 

Additionally, thanks to Blzut3, you may now download macOS binaries.

 

A change log is given later in this post.

 

One question that may be asked is what's the status of Android support. After all, the last release for Android preceded support for Wolf3D, thus Keen Dreams and the 3D Catacombs are the only covered games.

 

Thing is, the launcher and game controller support were a preparation for the Android port, at least partially. Later, it also involved adding general support for multi-touch input, preparing an Android project, and more. At the last, SDL2's Android port was already doing most of the important work, including a piece of Java code which is used in conjunction with Reflection Keen's and SDL2's C code. Still, work had to be done.

 

As of now, with the priority being lower, the work required for bringing Android support back will greatly involve the build system. I already migrated from Ant to Gradle back in 2019. This broke again after I transitioned from custom makefiles to CMake for the other platforms. In theory, using CMake and SDL2 altogether might actually make it easier to build for Android. However, there's still the issue that I have not just one, but multiple Android apps / activities to build (Keen Dreams, Catacombs, Wolfenstein 3D). Also, the general preference of mine, if Android support is ever resurrected, is to build shared SDL2 and Java code just once if possible. I don't know if there's a good way to do so while going through build system changes, though.

Jun 06, 2021 (v0.32.1):

* Redistributable application bundles can now be built for macOS with CMake.
(Thanks to Blzut3 for this addition.)
* Fix a possible crash, reproduced while binding an in-game action
to a controller button in the launcher with SDL 2.0.14 (thanks chungy).
This was a side-effect of an increase of SDL_CONTROLLER_BUTTON_MAX's value.
* More generally, mentions of enumerated controller buttons and axes
from SDL2 were replaced with equivalents defined in be_st.h. A few
related range checks were further added.
* The Xbox Elite controller's paddles can now be used as an additional
alternative to buttons, in case SDL 2.0.14 or compatible is used.
* Fix a significant usability problem with game controllers. When an
axis' value changes sign, a proper "release" notification should now
be sent for the original sign. Previously, the user could unexpectedly
get stuck in a specific state, like a continuously rotating protagonist.

 

[NY00123]

Version 0.33.0 is now available. Among other things, it has changes and additions related to compatibility (including limited forms of Vanilla Bug emulations).

Users of game controllers (via the "modern" controller scheme) should further pay attention, as other than one bug fix, there were technical changes related to internal game-specific key bindings.

 

Aug 02, 2021 (v0.33.0):

* Additional restructurings of game-agnostic code.
* Nuked OPL3 is updated to the last revision.
* Keen Dreams: Try to fix memory allocation problems
while using the data from the 2015 port.
* Keen Dreams, 3D Catacombs: For an uncached map, if the map header offset
being read is nonnegative but smaller than 8, just quit with an appropriate
error message. This fixes behaviors which greatly depend on the memory layout,
including a possible hang.
* Catacomb Abyss: Make it possible to warp to map 19, at least in general.
Also fix crashes upon attempts to load map 20 or 21 by
outputting a new error message.
* 3D Catacombs, Wolfenstein 3D: Add a "novert" toggle, letting the user
limit in-game mouse movements for turning/moving left and right.
* Wolfenstein 3D: Add a "lowfps" toggle, used for explicitly lowering the
frame rate. The purpose of this is controlling, to an extent, how far can
pushwalls move. Reason is, they were intended to move for 2 tiles, but
it turned out that they could move for 3 tiles as computers were
becoming faster. The Fake Hitler's fireballs also turned out
to move slower while the frame rate was higher.
* Wolfenstein 3D: Emulate, to an extent, the consequences of a
buffer overflow that follows a press on a pushwall while it's moving.
This is basically a Vanilla Bug, in which such walls are operated as if
they were doors, and unintentional changes to static objects may occur.
* Wolfenstein 3D: Upon trying to load a non-existent level
(say, in the March 1992 prototype), just report that the map isn't 64*64.
This is mostly similar to the outputs of the original executables,
while skipping potential undefined behaviors.
* Wolfenstein 3D: Fix menu navigation with the mouse while using a window.
* Change the way controller buttons get mapped under
the modern controller scheme. Basically, rather than parsing
the game's own configuration of keys and then using it as a base
for the mappings, just use the game's default bindings internally.
For instance, if a button was previously configured to be mapped
to "Jump" in Keen Dreams, it'll now map to the Ctrl key instead.
* Note that such buttons may internally get mapped to mouse buttons
instead of keyboard keys, in case "Motion emulation mode" is set to
"Mouse". The behaviors should remain unmodified in such a case.
* Further note that the above-mentioned changes impact
touch input, albeit the UI remains as-is.
* Amend controller usability fix from v0.32.1 to work better
when the setting "Motion Emulation Mode" is set to "Mouse".
* Add REFKEEN-NOTES.txt files to the source tree, describing technical changes
from the original versions of games, including cases in which such changes
exist for the purpose of emulating vanilla bugs. The information was
collected by looking for mentions of the word REFKEEN in the sources.
* Other miscellaneous fixes.

 

[NY00123]

Version 0.33.1 is now available. You may find it being called the "Loud Edition", if only due to default sound volumes being higher by default. This was a side-effect of a bug fix related to audio mixing.

 

Additionally, hopefully more accurate emulation of stereo panning should be available, based on information from a VOGONS forums post of James-F. I also added per-device volume controls.

 

Note that the volume controls are currently usable just before selecting a game to play (e.g., from the launcher), and not during gameplay. They're also done per-device, meaning, for instance, that you cannot separate the OPL music and sound volumes in Catacomb 3-D or Wolfenstein 3D.

 

A good question is if this follows Chocolate Doom well. I think it's a bit difficult to make a comparison, as Doom already provided its own in-game volume controls.

 

To explain why I think the added feature is acceptable for the Reflection Keen ports:
- First of all, most chances are the PC Speaker was really a separate device in the 90s. At least in theory, it could also have its own volume control.
- With the SB Pro and SB 16 (ISA) cards, you got a mixer in which you could control the volumes of OPL/FM/MIDI sounds/music and digitized sounds separately. If you asked Wolf3D to not detect an SB Pro, they would still be in effect.
- Otherwise, Wolf3D would tamper with these, basically setting the FM/MIDI volume to the max and changing digitized sounds' volumes for stereo panning. But tests done by Blzut3 showed to me that for Win9x, you could temporarily leave to the mixer program on Windows, change the FM/MIDI volume and then return to the game. Of course, this may have unexpected consequences related to stability, but it's probably expected.
- Even if the above weren't an option, someone probably used a sound card with volume controls independent of the ones that Wolf3D tampers with. Alternatively, maybe you could even use two separate cards: One for digitize sounds and another one for FM synthesis.

 

Aug 24, 2021 (v0.33.1):
* Identified as "Loud Edition" in-game, since changes related to audio mixing
increased the default sound volumes, at least outside of the PC Speaker.
* Nuked OPL3 is now at git revision bb5c8d08a85779c42b75c79d7b84f365a1b93b66.
* Introduce slider UI for a few launcher settings.
* Add a new Sound Blaster emulation option, currently relevant just for Wolf3D.
Use SB for mono output, and SB Pro or SB 16 for stereo panning of
digitized sounds. Note that it's more significant after choosing SB 16.
* Volumes for SB stereo panning, and internally also for OPL emulation
with relation to the SB Pro/16, are based on tables of dB values
posted here by James-F: https://www.vogons.org/viewtopic.php?t=54269
* Add user-accessible per-device volume controls, independent of what was
described earlier (e.g., Wolf3D's stereo panning). Note that these volumes
are configurable per-device.
* In particular, you cannot separate the OPL (AdLib) volumes
of music and sound effects in Catacomb 3-D or Wolfenstein 3D.
* Also, before playing Keen Dreams using the data from the 2015 port,
you should use the digitized sound volume slider. Reason is that
the sound effects were re-recorded into digitized forms.
* Balance the default PC Speaker volume, following the changes.
* Fix a sound mixing bug. Previously, the more sound sources were added to
the mixer, the quieter each of them would be in the mixed output. Instead
of taking the average of the sources, take the sum and clamp the result.
* Other miscellaneous changes.

 

[NY00123]

Hi,

 

As it should be known by now, Keen Dreams isn't the only game supported by Reflection Keen. Therefore, after close to 7 years, the project is being renamed. It's expected to be titled "ReflectionHLE" as of the following release.

 

Individual source ports might still be referred to as they have been, say "Reflection Keen Dreams". Additionally, the application icon will remain mostly the same. The main change will be the removal of the alphanumeric characters identifying the game, as a single executable is expected to support all games now. Given that they were all released under GPL-compatible terms, this should be feasible.

[NY00123]

Version 0.40.0 has been released. Since Keen Dreams isn't the only supported game, the project is being renamed as of this version. It's now titled "ReflectionHLE".

 

A side-effect of this is that you'll have to manually migrate saved games and settings. Hopefully, it shouldn't be more than copying a directory or two. See the top of the change log for details.

 

This version has the experimental addition of keyboard and mouse button overrides. They might not always work as expected. For instance, overridden keys may become unusable for cheat codes.

 

There's also an APK for Android again. Audio resampling isn't included in it, and there might be other problems related to audio. Generally speaking, the Android builds should be considered less supported.

 

Dec 31, 2021 (v0.40.0):

The project is renamed "ReflectionHLE" as of this version.
As a side-effect, users of earlier Reflection Keen versions will have to
manually migrate older settings and saved games. See paths given below.

Windows: %APPDATA%\reflection-keen => %APPDATA%\reflectionhle

macOS: ~/Library/Application Support/reflection-keen
=> ~/Library/Application Support/reflectionhle

Linux - two paths (assuming $XDG_CONFIG_HOME and $XDG_DATA_HOME aren't set):
~/.config/reflection-keen => ~/.config/reflectionhle
~/.local/share/reflection-keen => ~/.local/share/reflectionhle

* The project was renamed from "Reflection Keen" to "ReflectionHLE".
* A single executable can now be used for running the supported games.
It is identified as "ReflectionHLE".
* Due to using one exe, the application icon had its game identifiers removed.
* Additionally, the cfg file was split, with game-specific settings residing
in their own separate cfg files. Some cfg key and value names were further
renamed. Code was added for automatic migration of these from v0.33.1, but
this does not cover the directories which store the cfg files and saved games.
* The command-line options -skipintro and -showslides were removed. Instead,
you can show the intro or slides with sub-gamever selection via -gamever.
* Added the command-line option -listgamevers. It can be used for listing
supported game versions, instead of -?, which doesn't do so anymore.
* There's now experimental support for keyboard and mouse button overrides,
applying to in-game actions during gameplay.
* Note that these take a higher priority than in-game settings for the
same keys. They also have a higher priority than hardcoded behaviors of them.
For instance, it might be impossible to use a cheat code. You can still
use a cheat code via the on-screen debug keys, though.
* The on-screen keyboard with debug keys can now be displayed even
if there's no use of touch input or any game controller.
* Revert the impact of changes from v0.33.0 to the way controller buttons
get mapped under the modern controller scheme. Instead of mapping to the
default settings of a game, mapping is now done to the matching actions.
For instance, in Keen Dreams, you can map a button to Jump instead
of Ctrl (the default key for jumping in the original releases).
This is still done in a different manner from versions preceding 0.33.0,
as it's now less tied to the original games' keyboard settings.
* As a side-effect, there are changes to the impacts of the analog motion
toggles, even while using a D-pad. Additionally, the novert toggle should
also impact vertical motion from a game controller or touch input.
* Catacomb Abyss: Ensure it's possible to scroll through HELP.TXT using a game
controller or touch input, even if movement keys were changed in config.abs.
* General restructuring of the launcher's menus, following the support for
multiple games from a single exe and the addition of key/button overrides.
* Android builds are now possible again, albeit without audio resampling.
Generally speaking, these builds should be considered less supported.
* For technical reasons, sound output is done somewhat differently on Android.
There may still be problems with audio, including delays.
* Improvements to timing of sound playback during the intermission screen
in Wolfenstein 3D, Spear of Destiny and Super 3-D Noah's Ark. This
might be more noticeable while VSync is toggled on.
* Launcher control bindings for Wolfenstein 3D and Super 3-D Noah's Ark:
The weapons/feeders are now referred to by their numbers,
instead of game-specific descriptions.
* Emulation of mouse motion via game controllers was made
less frame rate dependent.
* Fix a typo impacting startup of Catacomb Apocalypse's hint book. The offset
of a textual screen as present in the original exe was wrong. The bug was
introduced during a refactor after the release of v0.30.1.
This screen is actually used just in the electronic catalog of the supported
shareware version of Catacomb Abyss, upon trying to print the catalog.
* Catacombs, Wolf3D: Fix building in case "char" is an unsigned type.
* Additional miscellaneous refactors, edits and fixes.

 

[Arno]

I appreciate the hard work you're putting into this source port. Maintenance can become quite a hassle with such a large catalogue of supported games and it's good to see that you give this a lot of thought.

 

On 12/31/2021 at 12:48 AM, NY00123 said:

* Fix a typo impacting startup of Catacomb Apocalypse's hint book. The offset
of a textual screen as present in the original exe was wrong. The bug was
introduced during a refactor after the release of v0.30.1.

 

This perfectly demonstrates the trade-off that is pretty common in software development. Refactoring benefits maintainability, but it comes with a risk of introducing unwanted side effects. In the long run refactoring always pays off, but it can be a challenge to verify with testing that everything is still working 100%.

 

Good luck with the road ahead for ReflectionHLE!

[NY00123]

A bit late reply, but I'm going to add one now:

 

On 1/1/2022 at 4:09 PM, Arno said:

I appreciate the hard work you're putting into this source port. Maintenance can become quite a hassle with such a large catalogue of supported games and it's good to see that you give this a lot of thought.

 

On 1/1/2022 at 4:09 PM, Arno said:

Good luck with the road ahead for ReflectionHLE!

 

 

Thanks for the support! It's true that supporting more games requires more work, which can feel repetitive at times. Surely, it obviously helps that the ports use a common codebase. I also made changes at times to make ReflectionHLE code more suitable for supporting multiple games. One example is the recent replacement of the command-line options -skipintro and -showslides with the ability to select a "sub-gamever" via -gamever. A list of the possible variations can be be shown with -listgamevers.

 

Also, as I wrote in the past, I generally prefer to not promise anything new before I'm quite sure that it'll be done. This also applies to support for other games, of course. On a comment which is more relevant to the post I'm replying to, each newly supported game is expected to require more work and time for future maintenance.

 

[NY00123]

Version 0.40.1 has been released. This update mostly consists of usability improvements. Also, for people less familiar with ReflectionHLE, a web page is now up: https://reflectionhle.com/

This release will hopefully resolve usability problems with keyboard overrides, and possibly also with other input devices. But there are great chances that there's a new bug not found by me, so bug reports will be welcome. You can use the GitHub repository's issue tracker, or report in the forums if you prefer so; Albeit I might not necessarily check all forums in which I posted about ReflectionHLE.
 

If you are upgrading from a version preceding 0.40.0, please check
migration notes for v0.40.0. The project was renamed to "ReflectionHLE"
at the time, and as a side-effect, older settings and saved games
have to be manually migrated.

Oct 14, 2022 (v0.40.1):

* Add a toggle for showing/hiding the ENDOOM screen on exit,
currently identified in the launcher as "Wait for input on exit".
Regardless of the toggle, it'll still appear in case of a nonzero exit code.
Other screens specific to shareware versions of certain games might
further keep appearing, or introduce their own delays.
* Updated search paths for Wolfenstein 3D and Spear of Destiny game data.
* Keyboard overrides don't fully replace the original uses of the keys
as of this version. For instance, if the launcher is used to select
the key W for moving forward in Catacomb 3-D, it can still be used
as a part of debug keys for warping to another level.
On the other hand, the familiar functionality of the key otherwise used
for the same in-game action will be disabled. To clarify for the example
of moving forward, the key generally used for this purpose from
Catacomb 3-D's own control panel (the Up arrow by default)
will not actually be used for moving forward.
Note that mouse button overrides are still full overrides and there's
no change in behaviors as describe above.
* Miscellaneous internal changes related to user input, related in part to
keyboard overrides. One example is a bug fix, where pressing on a key
moving Keen left, followed by another key moving Keen right, and then
releasing one of the keys, would make Keen stop moving altogether.
Also, as of this version, the analog motion is explicitly reserved for analog
controller axes; novert impacts these, as well as ordinary mouse movement.
* Use more locks for the audio mixer. This particularly
helped resolving unexpected slowdowns in Android builds
with BE_ST_MANAGE_INT_CALLS_SEPARATELY_FROM_AUDIO=1,
during playback of digitized sounds in Wolf3D.
* BE_ST_MANAGE_INT_CALLS_SEPARATELY_FROM_AUDIO=1 is used for Android again.
* Other updates related to Android builds.
* Made a few changes removing dependencies on libgcc and C++ libraries
as long as we don't actually need any of them.
* Other miscellaneous edits.