Are custom Doom64 maps/levels possible on real N64 hardware?

[orangefire222]

Hello all, been awhile since my last visit to doomworld.

 

I recently purchased an everdrive 64, and I'm wondering if there are any tools available for Doom64 modding? I've done a few google searches and looked around typical romhacking places, although not very thoroughly I'll admit, and haven't turned anything up.

 

I have some experience with doombuilder 64 back when Doom64EX came out, and I know it came with a tool to extract an iwad from the ROM.

 

Anyone know of any tools that can REinsert an edited iwad back into the ROM so I can load it up on the everdrive and play these on real hardware? Would love to see some custom content playing on the n64 on a TV.

[Chip]

I'm not a tech guy, so I am not totally sure, but I believe that it's impossible to do that on the hardware. I could be wrong, but it seems unlikely. also, the custom content you see probably is made in DOOM 2 or a non-official source port, so custom content from that would be impossible to plug into the real game.

[Doomkid]

Kaiser would be the man to ask about this sort of thing, though I imagine if extracting an iwad from a rom is possible, that inserting a compatible iwad (the reverse of the process, basically) should be trivial.

[Dexiaz]

Or ask the GEC Team

[Immorpher]

I have been keeping an eye out for a while, and there hasn't been any mods in terms of map swaps for the D64 ROM as far as I could find either. The closest I seen was some edits done on MAP1 on the ROM which did run in an emulator by @Erick194. But I know behind the scenes there has been work and progress on the D64 ROM side of things. I won't spoil anything as to give the people behind it a chance to reveal it on their own time. However, it will mean that even if it isn't possible to do full map swaps now, the future is looking bright in being able to do so.

[Dark Pulse]

My verdict: Technically possible, but would almost certainly need an expanded ROM - Doom has a 8 MB ROM and the game uses 7.75 MB of it.

 

Maps are also stored in a compression format that Kaiser has worked out how to decompress, but not recompress. The maps would thus need to be stored uncompressed, taking more space.

 

Erick does have a working compression (apparently), but the lumps are 1.06% bigger than the originals, and he can't figure out why. Even tried shooting John Carmack a tweet, and he replied saying he didn't know the format.

 

The man who WOULD know is Aaron Seeler, who works at Valve. And is, like, impossible to get a hold of for us plebs.

[sponge]

14 hours ago, Dark Pulse said:

The man who WOULD know is Aaron Seeler, who works at Valve. And is, like, impossible to get a hold of for us plebs.

 

And would also be dependent on him remembering code he wrote nearly a quarter century ago.

[Dark Pulse]

1 minute ago, sponge said:

And would also be dependent on him remembering code he wrote nearly a quarter century ago.

And even if he did, also dependent on him actually being willing to divulge it, which is no guarantee.

 

(Example: MAME's emulation of Primal Rage is imperfect due to inaccurate protection emulation, and they've tried to reach out to the developers to help make it improved. All the developers they reached out to either did not reply, or refused to help - so the protection remains imperfect until someone reverses it completely.)

[orangefire222]

Thanks for the replies guys.

 

Disappointing that there isn't much going on yet, I suppose I'll just have to wait and keep my eyes peeled.

[Shanes]

Kaiser told me that custom maps, for the actual ROM, would only be possible if you had the patch down to the exact bit size of the original ROM, or it would crash, and it may even if so..

[Dark Pulse]

16 hours ago, Shanes said:

Kaiser told me that custom maps, for the actual ROM, would only be possible if you had the patch down to the exact bit size of the original ROM, or it would crash, and it may even if so..

I don't recall him saying something like that, and even if that's the case, it's now moot with the reverse-engineered source.

[Immorpher]

Indeed, now with Doom 64 RE that Erick194 decompiled, bit sizes are no longer an issue. Right now it's more of building a tool set to work with Doom 64's compression algorithm, to compress the levels so that the Doom 64 code can read them. Or circumvent the compression algorithm in the code and load the levels uncompressed. Erick mentioned to me that there's some differences in terms of how textures are referred perhaps. But it seems like that Kaiser keeps the level format pretty authentic. I think if it runs on the remaster, unless it breaks a limit, it should theoretically run on the original code. I took a look with a buddy on the compression algorithm myself, but I have some learning to do to properly make a tool to utilize it.

[Dark Pulse]

1 hour ago, Immorpher said:

Indeed, now with Doom 64 RE that Erick194 decompiled, bit sizes are no longer an issue. Right now it's more of building a tool set to work with Doom 64's compression algorithm, to compress the levels so that the Doom 64 code can read them. Or circumvent the compression algorithm in the code and load the levels uncompressed. Erick mentioned to me that there's some differences in terms of how textures are referred perhaps. But it seems like that Kaiser keeps the level format pretty authentic. I think if it runs on the remaster, unless it breaks a limit, it should theoretically run on the original code. I took a look with a buddy on the compression algorithm myself, but I have some learning to do to properly make a tool to utilize it.

Well, the real magic trick would be figuring out that extra bit of compression, yes. IIRC Erick can produce compressed lumps that get read and are visually identical to the originals, but they're slightly bigger than the originals - that is, if you take a compressed lump, decompress it, and recompress it with Erick's method, it won't be 1:1 filesize/bytewise to the original compressed lump.

 

Figuring out how to make it identical would be the holy grail. But Carmack has said he doesn't know the compression (I asked), and the only other one who likely would, Aaron Seeler, is notoriously hidden and works for Valve now.

Edited June 8, 2021 by Dark Pulse

[Immorpher]

3 hours ago, Dark Pulse said:

IIRC Erick can produce compressed lumps that get read and are visually identical to the originals, but they're slightly bigger than the originals - that is, if you take a compressed lump, decompress it, and recompress it with Erick's method, it won't be 1:1 filesize/bytewise to the original compressed lump.

Indeed! And theoretically files from his compression algorithm should be fine in the Doom 64 ROM (even if not 1:1). So we have the code out there to do it. Just a matter of getting someone with the experience and passion to package it into a tool. I have the algorithms for both ways, but implementing them is a bit over my head at the moment.

[OpenRift]

This is a fascinating concept. If only we had the D64 source code...

[Wavy]

3 hours ago, OpenRift said:

This is a fascinating concept. If only we had the D64 source code...

Which we do!

[Shanes]

On 6/8/2021 at 12:38 PM, Immorpher said:

I don't recall him saying something like that, and even if that's the case, it's now moot with the reverse-engineered source.

 

It may be different now, that was years ago.  If I could get into my old account, which I think went dormant, I may be able to retrieve that PM..  Maybe it was that you couldn't add more, but he said he tried and it would crash.  Yes we do have the PC ports, but i would love to play custom maps on an everdrive..

 

Sorry, I hit the wrong quote, it was supposed to be a response to Dark Pulse..

Edited June 10, 2021 by Shanes

[Dark Pulse]

25 minutes ago, Shanes said:

It may be different now, that was years ago.  If I could get into my old account, which I think went dormant, I may be able to retrieve that PM..  Maybe it was that you couldn't add more, but he said he tried and it would crash.  Yes we do have the PC ports, but i would love to play custom maps on an everdrive..

Well in short, that's now possible, thanks to the reverse-engineered source. All you'd really need would be replacement maps and a toolchain. Compile the ROM and there's your new levels, byte-aligned and all.

[Shanes]

3 minutes ago, Dark Pulse said:

Well in short, that's now possible, thanks to the reverse-engineered source. All you'd really need would be replacement maps and a toolchain. Compile the ROM and there's your new levels, byte-aligned and all.

 

Great! I am out of the loop these days! Glad to hear its now possible :)  I didn't know of a reverse engineered source.  Last I read, something like the Doom 64 EX was a port using some reverse engineering, but only usable with a modified PC source code, on a PC..

[Dark Pulse]

2 minutes ago, Shanes said:

Great! I am out of the loop these days! Glad to hear its now possible :)  I didn't know of a reverse engineered source.  Last I read, something like the Doom 64 EX was a port using some reverse engineering, but only usable with a modified PC source code, on a PC..

Yeah, Wavy linked to the repository above.

 

https://github.com/Erick194/DOOM64-RE

 

People have already forked this and built some custom builds of Doom 64 with some features (like disabling the texture filtering, etc.)

[Immorpher]

2 minutes ago, Shanes said:

 

Great! I am out of the loop these days! Glad to hear its now possible :)  I didn't know of a reverse engineered source.  Last I read, something like the Doom 64 EX was a port using some reverse engineering, but only usable with a modified PC source code, on a PC..

Indeed and that's true too, Doom 64 EX was based on a partial reverse engineering of Doom 64. This was used to help create part of the reverse engineered code shared earlier. If anyone is curious and doesn't mind a self-plug, I wrote an article about the reverse engineering of Doom 64 here: 

 

[Shanes]

29 minutes ago, Dark Pulse said:

Yeah, Wavy linked to the repository above.

 

https://github.com/Erick194/DOOM64-RE

 

People have already forked this and built some custom builds of Doom 64 with some features (like disabling the texture filtering, etc.)

 

 

Thanks for the info!  I have been waiting a long time for those options, way out of my league..

[Blzut3]

On 6/8/2021 at 4:47 PM, Dark Pulse said:

Well, the real magic trick would be figuring out that extra bit of compression, yes. IIRC Erick can produce compressed lumps that get read and are visually identical to the originals, but they're slightly bigger than the originals - that is, if you take a compressed lump, decompress it, and recompress it with Erick's method, it won't be 1:1 filesize/bytewise to the original compressed lump.

 

Figuring out how to make it identical would be the holy grail. But Carmack has said he doesn't know the compression (I asked), and the only other one who likely would, Aaron Seeler, is notoriously hidden and works for Valve now.

If I recall my time spent researching compression correctly, there isn't a best algorithm for compressing input data into a bit stream.  I don't know what Erick's method is but if he's not well versed in writing compression code, it's likely just a greedy algorithm which will usually not produce the smallest possible output.  I don't think finding the exact algorithm to produce a 1:1 binary blob would be the holy grail, as it's not unlikely there are better, more exhaustive algorithms that can be used.  Not saying it is definitely possible to make a compressor better than the original, but it's not unlikely that the original was just something good enough while not dramatically increasing iteration times with computing power available at the time.  Having the original code would thus be mostly of academic use.

[Dark Pulse]

23 minutes ago, Blzut3 said:

If I recall my time spent researching compression correctly, there isn't a best algorithm for compressing input data into a bit stream.  I don't know what Erick's method is but if he's not well versed in writing compression code, it's likely just a greedy algorithm which will usually not produce the smallest possible output.  I don't think finding the exact algorithm to produce a 1:1 binary blob would be the holy grail, as it's not unlikely there are better, more exhaustive algorithms that can be used.  Not saying it is definitely possible to make a compressor better than the original, but it's not unlikely that the original was just something good enough while not dramatically increasing iteration times with computing power available at the time.  Having the original code would thus be mostly of academic use.

I mean, if better could be found, that would be even better, of course.

 

Mind you, this is really just being nitpicky about space savings. Obviously expanding a cartridge ROM is trivial stuff; expanding it to 64 MB would allow probably for dozens or even hundreds of uncompressed levels, at the cost perhaps of it being potentially slower to load. (No idea if it would take more time to load an uncompressed level due to needing to read more data, versus taking a compressed level and decompressing it.)

 

Still, with the source out there, it's now definitely possible for someone to take a crack at it if they really wanted to, and the decompression algorithm is known, so there's at least a vague idea of how the cart is doing it. From there it's just a matter of writing code that'd run nicely on the CPU (the NEC VR4300, which is essentially a MIPS R4300i core.)