Locating Duplicate Wads

I've got wads stored in various subdirectories on my hard drive that I've reaped from various resources. I'm sure I've got a ton of identical wads, probably different versions of the same one, but I know I've got at least a few handfuls of wads that are named alike but are definitely different. (i.e., there could be any number of CITY.WADs, or SEWERS.WADs, or BASE.WADs on the internet)

I'd like a program that would let me compare the dates and file sizes of similarly named wad files, and let me delete the ones that are identical and serving no purpose. Has anyone here encountered a similar issue? Any program recommendations?

On Linux, there's fslint that finds duplicate files... for another system, I guess you could just md5sum everything and find out what's identical.

Pretty much what chungy said. Get one of the gazillion freeware checksum apps out there, search your HD for *.wad, select all (except for zipped stuff) drag-drop it into the GUI, run the app, sort out the results by checksum and delete the dupes.

Using the Windows search function to search for *.wad and sorting by filesize or name or date isn't enough?

TimeOfDeath said:

Using the Windows search function to search for *.wad and sorting by filesize or name or date isn't enough?

That's...that's not how duplicate files work.

Marnetmar said:

That's...that's not how duplicate files work.

File size is good enough for most cases, and even if you want to be specific (ie. getting checksums or bit comparisons), file sizes help you narrow down the set quite a bit.

Yeah anything that uses filesystem metadata (name, size, timestamp, etc.) will be faster than a full MD5 or SHA checksum of every byte of that file. You can get further accuracy by using two pieces of metadata instead of just one, but that's probably only useful in cases where you can't do a real checksum (even something basic like CRC32).

Actually there was a recent similar thread, and it turns out that some versions of unzip (or equivalent tools that can handle ZIP archives) will display the embedded CRC32 of every included file inside a ZIP. But it gets messy because you have to parse all that and do the comparisons yourself.

If the files are already unzipped, it's probably easier to use an existing tool. There seems to be stuff like this for detecting duplicate image files (GIF, etc.) and they'll probably work equally well with any file format.

TimeOfDeath said:

Using the Windows search function to search for *.wad and sorting by filesize or name or date isn't enough?

Not always, downloading the same wad from different sources usually means mismatched time stamps and identical sized files aren't always the same. Binary file comparison or CRC checksums tend to be more reliable.

There are a ton of Windows-based utilities that can do duplicate file searches (and deletions) based on all of the above modes. Just google them.