Trying to improve the performance of my pk3 file

Hi guys, I'm trying to organize my map files well on a pk3 to improve performance. The map has always been strangely heavy, but as I currently only have 8 ram instead of 16 due to a memory that has burned, I need to know how to do this to run the map on pc.
 

I've been looking at some pk3 to see how to organize and I also saw an article on ZDoom, but I still don't know how to organize the maps inside the map folder. This seems simple when I simply create a map, but in this case the map has custom sounds and scripts, and I wanted to know how do I make these recognized within the folder and referred to their respective maps. Example; I have map1 that has a custom sound1, how do I reference this sound in a txt file so that it is used only in map1 while in the folder there are several other maps?? If map1 were an "actor" this would be as simple as referencing sounds to monsters.

Does anyone know how this is done or can show me an article?

That is not possible. ZDoom does not have a concept of map-specific resources. All resources that are loaded are equally available across all maps that are loaded.

Assuming that by "8 ram" you mean "8 gigabytes of RAM" and not "8 megabytes of RAM", that should be more than enough anyway.

This site on the ZDoom wiki explains how you set up a PK3. There's not really any such thing as "PK3 performance" - a PK3 is just a zip file, and the game unzips what it needs as you run it. If you are talking about FPS drops, it's nothing to do with your PK3.

8 hours ago, K_Doom said:

but I still don't know how to organize the maps inside the map folder.

Each map should be saved as its own .WAD file, containing only the relevant map files for that specific map. Typically that would be MAP01.wad, MAP02.wad etc. You would have nothing in those .wads except for the very specific lumps that make up the map data. No assets or graphics or textures or anything else.

8 hours ago, K_Doom said:

Example; I have map1 that has a custom sound1, how do I reference this sound in a txt file so that it is used only in map1 while in the folder there are several other maps

As Gez says, you can't. Every sound is accessible by every map. So when setting up your SNDINFO file, you simply give every sound you have a unique name, and then reference them accordingly in the maps. When you talk about a "sound1", are you meaning the ambient sound slot Thing? If so, you would need to set each separate sound up as a different ambient sound slot.

However, remember ambient sound Things are a holdover from early implementations of a speaker system, and these days something like PlaySound targeting a Mapspot gives you all the same degree of control as ambient sound thing without the limited number of slots.

1 hour ago, Bauul said:

As Gez says, you can't. Every sound is accessible by every map. So when setting up your SNDINFO file, you simply give every sound you have a unique name, and then reference them accordingly in the maps. When you talk about a "sound1", are you meaning the ambient sound slot Thing? If so, you would need to set each separate sound up as a different ambient sound slot.

Actually, you can. The engine allows additional SNDINFO files per map. Of course this won't reduce resource use - the feature was mainly added to allow building episodes out of single maps using the ambient sound feature.

Have you tried to create map by map, SLOWLY, and then merge it all with the order of maps, SLOWLY.

If it doesn't work, is probably obvious because I never edited so much maps in my life.

3 hours ago, Graf Zahl said:

 

Actually, you can. The engine allows additional SNDINFO files per map. Of course this won't reduce resource use - the feature was mainly added to allow building episodes out of single maps using the ambient sound feature.

 

That's interesting! How does it work? I can't find any documentation about this (at least on the SNDINFO ZDoom Wiki page), so I'd love to know more. If this is the case, OP's problem of having multiple "ambient sound slot 1"s all designed with different maps in mind would be solved instantly.

'No big magic involved.

MAPINFO has two per-map properties called "SNDINFO" and "SNDSEQ", there you can specify a file name that gets loaded on top of the global definitions when the map is loaded.

I added this feature many, many years ago so I could turn the KZDoom map series into an episode. Since each of these maps has its own SNDINFO my solution was to load them on demand.

TIL! Thanks Graf!

So @K_Doom, turns out yes you can define individual SNDINFO files per map. I'm just learning this myself, but I think the process would be:

1) For each map, create a new text file in your root PK3 directory, and name them something unique. 

2) Paste your ambient sound slot definitions for each map into the relevantly file you just created

3) In your MAPINFO lump, add the following property to each map: SndInfo = "your file name"

GZDoom will then load that file as a SNDINFO file just for that map. That way you can have individual ambient sound slot 1s for each map.

Try to compress the pk3 file with low of no compression, maybe this will help. 

8 hours ago, Gez said:

That is not possible. ZDoom does not have a concept of map-specific resources. All resources that are loaded are equally available across all maps that are loaded.

 

Assuming that by "8 ram" you mean "8 gigabytes of RAM" and not "8 megabytes of RAM", that should be more than enough anyway.

Yeah, 8 GB of ram should be enough but for some reason the file consumes so much ram that the computer crashes forcing me to restart it.

1 hour ago, Bauul said:

TIL! Thanks Graf!

 

So @K_Doom, turns out yes you can define individual SNDINFO files per map. I'm just learning this myself, but I think the process would be:

 

1) For each map, create a new text file in your root PK3 directory, and name them something unique. 

2) Paste your ambient sound slot definitions for each map into the relevantly file you just created

3) In your MAPINFO lump, add the following property to each map: SndInfo = "your file name"

 

GZDoom will then load that file as a SNDINFO file just for that map. That way you can have individual ambient sound slot 1s for each map.

I'm going to implement this, I hope it works since when I create the 'maps' folder and put my map there, GZDoom doesn't execute it. Showing the Entryway from Doom 2 instead.
Maybe I should organize all the files separately and then run them in the ZDL. This looks simpler and cleaner.

36 minutes ago, K_Doom said:

I'm going to implement this, I hope it works since when I create the 'maps' folder and put my map there, GZDoom doesn't execute it. Showing the Entryway from Doom 2 instead.

In a zip archive, maps in the maps folder are named according to their wad name, not to their internal slot name.

To make things clearer with an example: if you have this in your PK3:

maps\mymap.wad

and mymap.wad contains a MAP01 map.

Then your map slot will be "mymap", it will not be MAP01.

So you have to call the file maps\map01.wad instead.

6 hours ago, Gez said:

In a zip archive, maps in the maps folder are named according to their wad name, not to their internal slot name.

 

To make things clearer with an example: if you have this in your PK3:

maps\mymap.wad

and mymap.wad contains a MAP01 map.

 

Then your map slot will be "mymap", it will not be MAP01.

 

So you have to call the file maps\map01.wad instead.

Hi Gez, so I did as you told me and it really worked. But for some reason the sndinfo and sndseq files don't work... very strange.

9 hours ago, K_Doom said:

 

Yeah, 8 GB of ram should be enough but for some reason the file consumes so much ram that the computer crashes forcing me to restart it.

Keep in mind your OS and other programs are also using some of that RAM, so if you get very full, stuff will begin crashing.

10 minutes ago, Dark Pulse said:

Keep in mind your OS and other programs are also using some of that RAM, so if you get very full, stuff will begin crashing.

Yes I know, I spent the day looking at ways to improve ram usage and got good results. Even so, it's obviously not the same comfort I had when I still had 16GB of ram.
 

In fact, a tester complained about this to me, so I kept trying to improve the structure of my pk3 file containing the map. But as I had plenty of ram I left it for later, ironically now I have problems with heavy pk3 haha

Only so much you can do to "improve" a PK3 though, besides simply loading it up with less content.

The way it works is that it basically unzips it all into RAM, so if the uncompressed size would be that huge... well, there's your answer.

It's hard, the first version of my wad was like this, and it just crashed all of Slade3, and caused some delays in UDB.

The second version is this one, which crashed my computer yesterday and now I'm afraid to run too much haha (they also complained about consuming too much ram).

And currently it is like this, but there are several problems for GZDoom to recognize what I put in the folders, for example: it doesn't recognize GLDEFS for two actors of a monster.

I'm thinking of leaving it like this and running everything together in ZDL, I don't know if it's the best way... and I would also have to remove all the textures that I'm not going to use on the map so it doesn't get so heavy. However, there are more than 2600 textures... This is an example as I'm thinking of doing it now, in case I have more problems with the right pk3 structure form: 

  

When moving to a PK3 structure, there are a couple of things you can do about (technically duplicate) lumps that it's easy to have a lot of in a WAD file:

1. Consolidate them into a single large lump. For example, copy the contents of all your current GLDEFS lumps into a single GLDEFS lump.

2. Rename each lump in the style of "GLDEFS.Annihilator", "GLDEFS.Demolisher", etc. GZDoom shouldn't have any trouble parsing them as long as they have unique "extensions".

Personally I've done a bit of both; for example, I combined all of my custom monsters' SNDINFO files into a single lump named "SNDINFO.monsters", all the weapons' into "SNDINFO.weapons", and so on.

Check out my archive of test maps I made while I was learning to map - some are .wad file, but some are also based on simple .pk3 structure. Don't forget, you can run the UNZIPPED folder structure directly as well (drag and drop root of working directory):

https://github.com/smeghammer/snippets

So for added performance, try these:

 - load the unzipped folder hierarchy - that way he engine does no need to unzip at runtime.

 - if you DO use complied .pk3's make sure the zipping does not COMPRESS them (so the engine doesn't need to decompress as well as decompile)

 - make sure your PK3 directories only contain assets you actually need for your map(s) (don't load the entirety of Mek's Box'o'Skys or Otex...)

 - if there are ACS scripts, remove any redundant code (logging, unused functions/scripts, comments etc.)

On 6/21/2022 at 3:42 PM, K_Doom said:

The second version is this one, which crashed my computer yesterday and now I'm afraid to run too much haha (they also complained about consuming too much ram).

  

 

Yeah, having WAD files in the base of your PK3 causes them to load, but also causes the entire WAD to be loaded at startup, which isn't great for load times and reducing memory usage. Ideally, the WADs in your maps subfolder should contain ONLY map-specific data, with every other lump sitting in its appropriate place in the PK3, outside of a WAD.

What I'm seeing here shouldn't crash your machine, though. There's clearly something horribly wrong with your system, and I don't think GZDoom or your PK3 is causing it.

On 6/27/2022 at 11:05 AM, Kinsie said:

Yeah, having WAD files in the base of your PK3 causes them to load, but also causes the entire WAD to be loaded at startup, which isn't great for load times and reducing memory usage. Ideally, the WADs in your maps subfolder should contain ONLY map-specific data, with every other lump sitting in its appropriate place in the PK3, outside of a WAD.

 

What I'm seeing here shouldn't crash your machine, though. There's clearly something horribly wrong with your system, and I don't think GZDoom or your PK3 is causing it.

You're right, the computer is having problems and that's why I took it to fix it. Although the structure of my pk3 file is not ideal, my computer has only had problems with it recently. Other than that only one tester complained about it, so I thought the problem was pk3.

On 6/26/2022 at 5:17 PM, smeghammer said:

Check out my archive of test maps I made while I was learning to map - some are .wad file, but some are also based on simple .pk3 structure. Don't forget, you can run the UNZIPPED folder structure directly as well (drag and drop root of working directory):

 

https://github.com/smeghammer/snippets

 

So for added performance, try these:

 

 - load the unzipped folder hierarchy - that way he engine does no need to unzip at runtime.

 - if you DO use complied .pk3's make sure the zipping does not COMPRESS them (so the engine doesn't need to decompress as well as decompile)

 - make sure your PK3 directories only contain assets you actually need for your map(s) (don't load the entirety of Mek's Box'o'Skys or Otex...)

 - if there are ACS scripts, remove any redundant code (logging, unused functions/scripts, comments etc.)

Thank you, seeing projects from more experienced people than me have always helped me. It was even through watching tutorials and projects from others that I learned everything I know.

My only question is how to leave in the file only the textures that I will use in the map and exclude the others? For example OTEX easily has over a thousand textures, how do I remove so many textures that I won't use in my map?

Textures: yeah - get the zip version of otex and  copy just the ones you need (including skies) into the /textures/ namespace of your working pk3 directory. 

It was hard to understand this at first but once you realise that 'lumps' can be thought of as files (graphics, sounds, text files etc.) and that a pk3 file is just a zipped folder hierarchy it suddenly clicked. The otex wad is just a collection of lumps - i.e. graphics files. You could of course use Slade3 to extract what you want but as it is already available as a zip archive, that is just easier. 

Take my Belial's Keep project - there are a few custom textures in this (not otex, but the principle is the same) and all I do is place them in the above directory. GZDoom 'knows' that graphics in here are textures that can be referred to in maps.