Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
invictius

How much more compression could I get from zipping zipped wads?

Recommended Posts

Got 1.1tb of wads and I want to back them up on ideally a 500gb drive.  Some are even unzipped.  How much can I expect from zipping the unzipped ones, also?

Share this post


Link to post

Compressing something that's already compressed usually gains nothing and can even make the files larger sometimes.  You would probably make better gains by unzipping all the .ZIP files and then rezipping them in a single unit so that the compression algorithm can make better gains on elements that are common across files, although there are imaginable reasons you might not want to do that (like preserving the original .ZIPs for historical archival or the like).

 

However, there is something to be gained in terms of disk space by packing a zillion small files into one big one, because of the way storage works.  Clusters have a certain size that gets used up even if a file's data doesn't fully occupy them.  I believe the standard cluster size is 4 kilobytes so you might gain something like 1 to 3K per file by packing up thousands of files into one.

 

For what you're considering, that probably wouldn't be enough, and besides there are also limits on how large a single file is allowed to be.  If you unpacked everything into sorted folders, and then repacked each folder as something more efficient like a .7Z, it might be possible, but what you describe is looking for a 50% reduction in size which is a lot to ask.

Edited by ETTiNGRiNDER : Address the specific question a bit more thoroughly

Share this post


Link to post
11 minutes ago, ETTiNGRiNDER said:

You would probably make better gains by unzipping all the .ZIP files and then rezipping them in a single unit so that the compression algorithm can make better gains on elements that are common across files

Zip doesn't support that feature; all files are compressed by themselves and similarity with other files isn't considered.

 

1.1T into 500G... that sounds tricky indeed, and with that kind of storage in the first place, I'd guess you have a lot of WADs with already-compressed data lumps (eg, MP3 and Vorbis). I might try seeing if wimlib might help you, since it will only store identical files once in the whole archive (a command like "wimcapture E:\source\wads F:\archive.wim" would be all you need to do)

Share this post


Link to post

How the hell did you end up with 1.1 TB of Doom wads? Mine is like less than 5 GB.

Share this post


Link to post
15 hours ago, chungy said:

Zip doesn't support that feature; all files are compressed by themselves and similarity with other files isn't considered.

WinRAR does. The option is "Create solid archive". Here's what the documentation says:

Solid archive is an archive packed with a special compression method, which treats several or all files within the archive as one continuous data stream. WinRAR supports solid mode only in RAR archiving format, ZIP archives are always non-solid. Use Create solid archive option in archiving dialog or -s command line switch to enable solid archiving.

 

I don't know if 7-Zip has something similar. But, yeah, you might get 1.1Tb into 500Gb if all the files are uncompressed, but, if the bulk is already zipped, probably not. Compression works when files have duplicated blocks in them. Most do. But, once zipped, there's not much duplication left to exploit.

 

I'd make 3 suggestions:

1. Get a program that searches for exact duplicates, and prune out any duplicate files. You might find that you have a bunch of those.

2. Try various compression technologies, as each usually has different results. 7-Zip is supposed to be really good, but the amount of compression is really based on the contents of the files.

3. External HDDs are cheaper than ever. If big enough, you might be able to get multiple backups to fit.

 

3 hours ago, Nevander said:

How the hell did you end up with 1.1 TB of Doom wads? Mine is like less than 5 GB.

WADs with embedded mp3 songs tend to be huge. Also, there's lots of duplicate files out there, even with different filenames. Still, that's a lot of Doom content!

Share this post


Link to post
1 hour ago, kb1 said:

WinRAR does. The option is "Create solid archive". Here's what the documentation says:

Solid archive is an archive packed with a special compression method, which treats several or all files within the archive as one continuous data stream. WinRAR supports solid mode only in RAR archiving format, ZIP archives are always non-solid. Use Create solid archive option in archiving dialog or -s command line switch to enable solid archiving.

Yeah, read the paragraph you quoted again. I was talking about the Zip format, which doesn't support a solid mode. I know 7z and RAR do (as well as the WIM format I recommended early, but you'd have to pass in the --solid parameter).

 

In addition, the WIM format already takes care of your suggestion #1 :)

Share this post


Link to post
18 minutes ago, Cacodemon345 said:

Isn't WIM format some sort of packed-up Windows Installation file?

No.

Share this post


Link to post

It's used by the installation process, yes, but it's also a suitably generic archive format that can be used for anything you want.

Share this post


Link to post
10 hours ago, Nevander said:

How the hell did you end up with 1.1 TB of Doom wads? Mine is like less than 5 GB.

I backed up all those sites that host wads (when you fire up zandronum/whatever server browser and it finds the wad)

Share this post


Link to post

In general you can't compress files which are already compressed. Compression algorithms work by looking for patterns in data, and compressed data looks random. That also goes for formats like MP3, which are already compressed.

 

What you might want to consider though is that the ZIP format supports many different compression algorithms, and some of the newer algorithms are much better than the older ones. If you have ZIP files you've downloaded from idgames for example which were made back in the '90s, you might find that uncompressing and recompressing them with a modern ZIP program gives better results.

Share this post


Link to post
7 hours ago, fraggle said:

What you might want to consider though is that the ZIP format supports many different compression algorithms, and some of the newer algorithms are much better than the older ones. If you have ZIP files you've downloaded from idgames for example which were made back in the '90s, you might find that uncompressing and recompressing them with a modern ZIP program gives better results.

 

By modern, do you mean Deflate (type 8)?  I didn't think idgames had that many non-deflated archives.

 

And if you didn't mean Deflate, I think it would be wise to add, parenthetically, that not all tools that deal with ZIP files know how to deal with the myriad of strange compression options that ZIP supports.  I know both ZDoom and SLADE warn about such things.

Share this post


Link to post
4 hours ago, AlexMax said:

By modern, do you mean Deflate (type 8)?  I didn't think idgames had that many non-deflated archives.

Modern tools even support methods like bzip2.

 

4 hours ago, AlexMax said:

 

not all tools that deal with ZIP files know how to deal with the myriad of strange compression options that ZIP supports.  I know both ZDoom and SLADE warn about such things.

I wasn't aware of that. I guess it might be an issue if you want to play WADs directly from ZIPs (I think ZDoom supports that?) but if it's just to save disk space then it's probably fine.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×