/idgames database ID numbers

Sorry, a little off-topic. When an idgames entry gets deleted (I think it says something like "id not valid" or something like that), does another wad/file takes its place, or is it never used again? For example, if id=5000 is deleted, will any other wad/file will take its place as the "new" 5000 or it won't be used again?

I expect it's almost certainly the case that an ID will never be reused, but if you're building anything around that assumption you're probably making a mistake.

You should probably never rely solely on the "id" assigned to a file. The system supports looking up a file by its directory and filename, e.g. http://www.doomworld.com/idgames/index.php?id=themes/xmas/xmasdoom.zip

Taking advantage of this small bump to express my deepest thanks to chungy for offering this small but useful contribution, I downloaded the entirety of it, and I'll be seeing it 'till the end of times.

Bloodshedder said:

You should probably never rely solely on the "id" assigned to a file. The system supports looking up a file by its directory and filename, e.g. http://www.doomworld.com/idgames/index.php?id=themes/xmas/xmasdoom.zip

Does that mean that using an idgames://<file ID> URL to refer to a file shouldn't be relied on as well?

spicyjack said:

Does that mean that using an idgames:// URL to refer to a file shouldn't be relied on as well?

I don't write that part, so you'll have to ask Matt if he's added support for the file syntax.

Bloodshedder said:

I don't write that part, so you'll have to ask Matt if he's added support for the file syntax.

No, sorry, my question was for using the idgames:// URL syntax with a file ID, for example, idgames://12345.

If file IDs can change in the future, what's the point of referring to files by file ID in an idgames:// URL? Will you get some kind of redirect to another idgames:// URL if the file ID changes at a later date, or ???

I've noticed that idgames:// URLs are not generated properly if "id" is not used as an argument for the page. I was not aware that "file" was a possible parameter for idgames/index.php, which is something I'll have to add to the GET action for the API.

However I handle it, the policy for idgames://15156 and idgames://levels/doom2/Ports/megawads/cchest3.zip is that they should retrieve the same file (since you would have to use GET to get the file data for both). Of course, this is still entirely up to the interpreting program.


The entire purpose for idgames:// was for telling outside programs what file to retrieve from an idGames Archive, similar to how aim:// or steam:// or other third-party protocols are interpreted - by an associated program that understands what to do with it.

spicyjack said:

If file IDs can change in the future, what's the point of referring to files by file ID in an idgames:// URL? Will you get some kind of redirect to another idgames:// URL if the file ID changes at a later date, or ???

Ideally the IDs will never change. But if the database were to be lost and had to be regenerated, they would be different. Hopefully this will never happen.

MTrop said:

I've noticed that idgames:// URLs are not generated properly if "id" is not used as an argument for the page. I was not aware that "file" was a possible parameter for idgames/index.php, which is something I'll have to add to the GET action for the API.

I actually fixed this a day or two ago.

Zed said:

Sorry, a little off-topic. When an idgames entry gets deleted (I think it says something like "id not valid" or something like that), does another wad/file takes its place, or is it never used again? For example, if id=5000 is deleted, will any other wad/file will take its place as the "new" 5000 or it won't be used again?

To actually answer this original question, auto_increment values are not normally re-used, so a deleted file's ID should never be replaced by another file.

Bloodshedder said:

You should probably never rely solely on the "id" assigned to a file. The system supports looking up a file by its directory and filename, e.g. http://www.doomworld.com/idgames/index.php?id=themes/xmas/xmasdoom.zip

I asked for this a while back and didn't realise my wish had been granted. Thanks!

Ideally I wish we would just eliminate the notion of "idgames ID" entirely as a public thing. Change eg. ?id=8808 to redirect to ?id=levels/doom2/deathmatch/d-f/deca.zip, replace all the links, etc.

The current system encourages the use of these numbers (there's even the "idgames protocol" link on the page). They're meaningless magic numbers that aren't even guaranteed to be stable and I wish they'd go away.

fraggle said:

Ideally I wish we would just eliminate the notion of "idgames ID" entirely as a public thing. Change eg. ?id=8808 to redirect to ?id=levels/doom2/deathmatch/d-f/deca.zip, replace all the links, etc.

I have been planning on doing this for a few months now. I don't yet know when it will happen.

fraggle said:

Ideally I wish we would just eliminate the notion of "idgames ID" entirely as a public thing. Change eg. ?id=8808 to redirect to ?id=levels/doom2/deathmatch/d-f/deca.zip, replace all the links, etc.

The current system encourages the use of these numbers (there's even the "idgames protocol" link on the page). They're meaningless magic numbers that aren't even guaranteed to be stable and I wish they'd go away.

Can it be replaced with some kind of URL shortening service then? A read-only by end user service that can generate the same URL every time, and not just to /idgames, but to the shovelware CDs stored on archive.org for example, in case someone actually does go through them and catalog them at some point. Or any other 3rd party site with WADs. There was another thread where someone had a giant list of sites they were going through and downloading.

For URL shortening, I was thinking something along the lines of generating a checksum from domain + path + filename, turn that in to base32, and the first files indexed get the shortest URLs possible, longer URLs are generated from the hashes that are generated in order to prevent hash collisions. For /idgames files, use fullsort.gz, to generate URLs from oldest file to newest.

Edit: added mention of fullsort.gz for determining order for indexing of /idgames

All internal page URLs on /idgames should no longer be using id numbers now, except for the idgames:// link which I haven't changed yet. The new parameter is file= but specifying a path and filename with id= will also work, but isn't recommended. file is the preferred parameter.

Welp, looks like I need to update my client programs...

EDIT: Also, I'll need to update the API.

MTrop said:

Welp, looks like I need to update my client programs...

EDIT: Also, I'll need to update the API.

If the automatic redirection is breaking the API, it can be disabled for now.

EDIT: By automatic redirection, I mean that any URL using 'id' got redirected to use 'file'. That part has been disabled for now.

Oh no - it's not broken at all.

I'll just have to add support for GET using "file" as well as "id", just to keep things consistent.

Okay, new version sent to Bloodshedder.

If the version is 2 and the "updated" date is 1/6/2014, it has been applied.

fraggle said:

They're meaningless magic numbers that aren't even guaranteed to be stable and I wish they'd go away.

Yeah I'm not real proud of the system it ended up with. I don't even remember why I decided to do it with magic numbers instead of just passing the relative directory structure. I think it was probably some idea that somehow only integers could be a primary key and therefore looked up faster in the MySQL database, as if that would at all be the bottleneck.

The /idgames Database is, like, my eternal reminder that mistakes you make in a day (I coded the entire thing in like one 8 hour binge) can haunt people a decade later.

Linguica said:

Yeah I'm not real proud of the system it ended up with. I don't even remember why I decided to do it with magic numbers instead of just passing the relative directory structure. I think it was probably some idea that somehow only integers could be a primary key and therefore looked up faster in the MySQL database, as if that would at all be the bottleneck.

From a database design standpoint, a surrogate auto-incrementing primary key is actually a better decision than making a text field the primary key.

The /idgames Database is, like, my eternal reminder that mistakes you make in a day (I coded the entire thing in like one 8 hour binge) can haunt people a decade later.

I don't think it's that terrible. Sure, it could be done much better today, if I had the inclination or skills...