Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
lucius

The Force Engine (Jedi Engine Port/Replacement)

Recommended Posts

Thanks for taking the time to put together that detailed blog post on 3D object rendering. I always like to read about how various 3D effects were achieved under strict performance limitations.

Share this post


Link to post
5 hours ago, Arno said:

Thanks for taking the time to put together that detailed blog post on 3D object rendering. I always like to read about how various 3D effects were achieved under strict performance limitations.

I'm glad you liked that type of content. :)

 

At some point, I have to continue the Dark Forces DOS rendering series.

 

@VGA - I recently added GIF recording to the engine, so I decided to use it to show the 3D object rendering in the Classic Renderer, as you see here. So it is now almost as easy to record GIFs and it is to take screenshots.

Share this post


Link to post
10 hours ago, lucius said:

I'm glad you liked that type of content. :)

 

At some point, I have to continue the Dark Forces DOS rendering series.

 

@VGA - I recently added GIF recording to the engine, so I decided to use it to show the 3D object rendering in the Classic Renderer, as you see here. So it is now almost as easy to record GIFs and it is to take screenshots.

Ah, nice. But isn't it big? The filesize for a few seconds of gif action? I am not at my PC so I can't easily check the dimensions and such.

 

Also, I realised I buried your update post in the previous page, here it is again LOL

 

On 12/27/2020 at 11:52 PM, lucius said:

I posted a final blog entry for 2020, the Classic Renderer is finally complete - just in time to end 2020.

 

TFE Classic Renderer Complete


The 3D Object rendering system, derived from the original DOS code, is now working in both 320x200 (fixed-point sub-renderer) and at higher resolutions and in widescreen (floating-point sub-renderer). This completes the Classic Renderer reverse-engineering and integration project, allowing me to move on to reverse-engineering the INF system.

 

. . . .

 

Read the post here: https://theforceengine.github.io/2020/12/27/TFE-Classic-Renderer-Complete.html

 

Share this post


Link to post

GIFs can be big, yes. But it is nice for some quick animations that play well in the browser - great for forum or blog posts. Obviously, GIF recording is no replacement for proper videos. Currently, the engine is set to record at 15fps and I usually record them at 640x480 in order to manage the size.

Share this post


Link to post

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

Share this post


Link to post
15 minutes ago, Annunakitty said:

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

Lucius has said that he's entirely focusing on Jedi Engine for now.

Share this post


Link to post
18 minutes ago, Annunakitty said:

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

 

Have you seen Daggerfall Unity? It's going to be amazing. https://www.dfworkshop.net/

Share this post


Link to post
3 hours ago, Annunakitty said:

Oh shiiiiiiiiii, I was really worried that XLEngine was going to be dead for good but it's nice to know that at least some part of it lives on in this!!!  Any chance that Daggerfall port is still cooking at all or are you focusing on Jedi for now?

The Force Engine is focused on the Jedi Engine games - so Dark Forces and then Outlaws. The Force Engine will remain a focused project and I will not be adding support for any other games,etc. until Dark Forces and Outlaws are completely playable and the modding tools are ready. And for these kinds of projects, TFE will remain the sole focus - I don't have enough bandwidth for multiple large-scale projects at once.

 

I haven't given up on the possible idea of a proper reverse-engineered "X Engine" port in the future (Future Shock, Daggerfall, etc.) - but that is a different project and in a distant future (if it happens). That said, I think Daggerfall Unity will work well for you. :)

Edited by lucius

Share this post


Link to post

TFE Update and 2021 Plans


As the first blog post of 2021, this will cover several topics. First, an update regarding the progress of The Force Engine. That will be followed by TFE plans for 2021 - the year of the first fully playable release (hopefully), and finally the first “experimental” build and what that is all about.

. . . .

 

Read the post here: https://theforceengine.github.io/2021/01/18/TFE-Update-2021-Plans.html

Share this post


Link to post

INF System and TFE Progress

 

In the last post - TFE Update and 2021 Plans - I talked about the INF & Classic Renderer release which was to be followed by a “Player Control & Physics” release later in the year. Due to the way the renderer, INF and player controller are connected there has been a change of near-term plans. This post will go over those changes, talk about how the INF system works internally and provide a general update regarding the progress towards this year’s goal of reaching the 1.0 release.

. . . .

Read the rest of the post here: https://theforceengine.github.io/2021/03/02/InfSystem.html

Share this post


Link to post

Oh wow, I didn't even know the XL engine was still a thing. I remember religiously checking the forums everyday for progress on the XL port of Daggerfall around 2013-2014 until it just went silent for a while and then Daggerfall Unity started to catch some steam. Glad to know that the XL engine is still a thing, I'm guessing the Daggerfall and Outlaws ports are a thing of the past now? There was another port that was being worked on too but I forget what it was, might have been another Lucasarts fps?

Share this post


Link to post
2 minutes ago, Doominator2 said:

Oh wow, I didn't even know the XL engine was still a thing. I remember religiously checking the forums everyday for progress on the XL port of Daggerfall around 2013-2014 until it just went silent for a while and then Daggerfall Unity started to catch some steam. Glad to know that the XL engine is still a thing, I'm guessing the Daggerfall and Outlaws ports are a thing of the past now? There was another port that was being worked on too but I forget what it was, might have been another Lucasarts fps?

You might want to read https://theforceengine.github.io/ and https://theforceengine.github.io/2020/05/05/firstpost.html to get an idea what The Force Engine is. You can see the blog archive here: https://theforceengine.github.io/archive.html.

 

In summary, TFE is a fresh start, focused on accuracy using reverse-engineering, and is focused only on "Jedi Engine" games - Dark Forces and Outlaws. Dark Forces is the current focus but Outlaws shares a lot of the same code. The goal is for TFE to be a complete DosBox replacement for Dark Forces by the end of the year (meaning it is accurate to the original but does not use emulation).

Share this post


Link to post
Quote

A Note on Timing

Dark Forces quantizes time to 145 ticks per second.

Interesting. I wonder why the LucasArt devs chose this value -- I know Doom's value of 35 ticks per second was based on video refresh rate on the usual hardware of the era (70 Hz for 320x200 in 256 colors), as a simple way to keep game updates and video updates in sync.

Share this post


Link to post
1 hour ago, Gez said:

Interesting. I wonder why the LucasArt devs chose this value -- I know Doom's value of 35 ticks per second was based on video refresh rate on the usual hardware of the era (70 Hz for 320x200 in 256 colors), as a simple way to keep game updates and video updates in sync.

The strange thing is that Dark Forces uses the same 70Hz video mode. I think the reason it isn't tied directly to framerate is that the Jedi Engine uses a "Task" system, where tasks can yield, run at specific rates, etc.. Anyway, game and INF code is split into these tasks with the idea that not everything needs to run every frame. In the case of the INF system, it doesn't really work - I mean the main update task runs every frame in almost every map, as does the texture animation task, but I think this is one of the reasons they chose more granular timer settings. For TFE, this is being greatly simplified since it doesn't really buy anything but the timing itself has to stay the same.

 

That said, I didn't believe the 145 ticks / second thing at first, thinking I made a mistake. But you see it over and over again in the code, it is no mistake as odd as it is. :)

Share this post


Link to post

Closing the Game Loop

 

The previous blog post talked about refactoring the code so that the reverse-engineered INF system, collision system, and renderer can all use the original, fixed-point level data. Another goal was to enable caching for systems, like the floating-point high resolution renderer, that need to transform the data in some way. It illustrated the connected nature of the core game and engine code

. . . .

Read the post here: https://theforceengine.github.io/2021/04/12/GameLoop.html

Share this post


Link to post

Very cool! For possible future project news posts, I'm curious to know what your process for reverse engineering is. Do you use a disassembler, monitor the running game in DOSBox, something else, or a mix of techniques?

I've never done this kind of programming before and it's where lots of "reimplement old game" open source projects seem to hit a hard wall after getting the formats figured out and the world rendering.

Share this post


Link to post
19 hours ago, JPL said:

Very cool! For possible future project news posts, I'm curious to know what your process for reverse engineering is. Do you use a disassembler, monitor the running game in DOSBox, something else, or a mix of techniques?

I've never done this kind of programming before and it's where lots of "reimplement old game" open source projects seem to hit a hard wall after getting the formats figured out and the world rendering.

Honestly, there is no magic to it - just a lot of tedious work, reading documentation, and puzzle-solving. :)

 

It is basically a combination of disassembly + debugging in DosBox (for DOS-based games anyway, you can use Visual Studio for Windows programs). Most of the work is figuring out patterns, keeping track of all of your functions, variables, and structures, and constantly updating them with good names and types as you figure things out.

 

For example, take this function:


void obj_setupAnimation()
{
    Logic* logic = s_curLogic;    // eax
    logic->u68 = (logic->u68 | 1) & 0xfd;
    logic->nextTick = s_curTime + logic->delay;

    SecObject* obj = logic->obj;    // ebx
    obj->anim = obj_getAnimationIndex(5);    // eax -> edx
    obj->frame = 0;
}

 

You can see that most elements of the logic structure have already been figured out but 'u68' is unknown. So I have the name stubbed out (u = unknown, 68 = at offset 0x68 in the structure). Later I will see logic->u68 in action and figure out what it is used for, then I will go back and update the name. By updating the name, that will enable new understanding in existing code which will allow me to figure out other elements of the code (for example '1' is likely a flag, what does it do? Why are some flag bits cleared (the 0xfd)).

 

Another big component, once you figure what the low-level code is doing is figuring out why. For example, take this function:


s32 vec2ToAngle(fixed16_16 dx, fixed16_16 dz)
{
        if (dx == 0 && dz == 0)
        {
            return 0;
        }

        const s32 signsDiff = (signV2A(dx) != signV2A(dz)) ? 1 : 0;
        // Splits the view into 4 quadrants, 0 - 3:
        // 1 | 0
        // -----
        // 2 | 3
        const s32 quadrant = (dz < 0 ? 2 : 0) + signsDiff;

        // Further splits the quadrants into sub-quadrants:
        // \2|1/
        // 3\|/0
        //---*---
        // 4/|\7
        // /5|6\
            //
        dx = abs(dx);
        dz = abs(dz);
        const s32 subquadrant = quadrant * 2 + ((dx < dz) ? (1 - signsDiff) : signsDiff);

        // true in sub-quadrants: 0, 3, 4, 7; where dz tends towards 0.
        if ((subquadrant - 1) & 2)
        {
            // The original code did the "3 xor" trick to swap dx and dz.
            std::swap(dx, dz);
        }

        // next compute |dx| / |dz|, which will be a value from 0.0 to 1.0
        fixed16_16 dXdZ = div16(dx, dz);
        if (subquadrant & 1)
        {
            // invert the ratio in sub-quadrants 1, 3, 5, 7 to maintain the correct direction.
            dXdZ = ONE_16 - dXdZ;
        }

        // subquadrantF is based on the sub-quadrant, essentially fixed16(subquadrant)
        // which has a range of 0 to 7.0 (see above).
        const fixed16_16 subquadrantF = intToFixed16(subquadrant);
        // angle = (int(2.0 - (zF + dXdZ)) * 2048) & 16383
        // this flips the angle so that straight up (dx = 0, dz > 0) is 0, right is 90, down is 180.
        const s32 angle = (2 * ONE_16 - (subquadrantF + dXdZ)) >> 5;
        // the final angle will be in the range of 0 - 16383
        return angle & 0x3fff;
}

Where:

signV2A is: signV2A(x) { return (x < 0 ? 1 : 0); }

 

You can imagine, before the comments and some functionally equivalent changes this looked like a string of random math operations. Some things, like abs() hide a lot of underlying complexity in the assembly. But if you want to be able to fix bugs and extend the code in the future it is not enough to figure out what the code is doing, but you need to figure out why it is doing it. So having a good background in game/engine programming helps and of course some knowledge of assembly, though even then you will have to look up some of the more esoteric instructions every now and again.

 

So it is a mix of tedium with spikes of puzzle-solving and head-scratching. After a while, you just start reading the assembly and just start typing the equivalent C-code. :)

Share this post


Link to post

Thanks for the explanation Lucius! I'm familiar with that style of gradually filling in a disassembly. Your commenting style is very good and makes it very easy to understand what's going on in that excerpt, I strive for that with my own work. Looking forward to playing the results, keep up the great work.

Share this post


Link to post
1 hour ago, JPL said:

Thanks for the explanation Lucius! I'm familiar with that style of gradually filling in a disassembly. Your commenting style is very good and makes it very easy to understand what's going on in that excerpt, I strive for that with my own work. Looking forward to playing the results, keep up the great work.

Thanks. To be clear, though, comments in most of the code aren't as dense. Generally, it is preferable to write "self-documenting" code and be a little more sparse with comments (since comments can go out of date over time and reduce the amount of code you can see at once). However, comments should always be used when the "how" and "why" of the code are non-obvious, as is the case with that particular function.

Share this post


Link to post
21 hours ago, lucius said:

Generally, it is preferable to write "self-documenting" code and be a little more sparse with comments (since comments can go out of date over time and reduce the amount of code you can see at once).

This is a good practice and one of the software engineering guidelines that Robert C. Martin a.k.a. "Uncle Bob" advocates in his book Clean Code. That book was an eye-opener for me.

But indeed, your code sample is the kind of algorithm where comments do have value.

Share this post


Link to post

@Dzierzan: That voxel model of Palpatine is phenomenal! The smooth weapon-firing animations are cool too.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×