Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
MTrop

/idgames Archive Database REST API Now Available

Recommended Posts

spicyjack said:

I'll keep in mind the contact e-mail address mentioned above in the about call, however, any chance of posting the API script somewhere publicly, preferably some place with a issues/bug queue, like GitHub/Bitbucket/Sourceforge/etc.?

Thanks again!


No prob.

Unfortunately, the code will not be posted publicly, as Bloodshedder is a bit protective of code that would show off some of Doomworld's database layout, but feel free to E-mail me about any issues that may come up!

Share this post


Link to post

I will marry the first person to code a search feature that allows you to search for names shorter than three letters. :V

Share this post


Link to post

(Note: I also e-mailed this directly to MTrop, but posting it here so everyone can see it)

I've been parsing the contents of the idGames API over the last couple of weeks, and I found some more issues with the JSON. The XML requests are parsing fine, so no worries there.

My methodology was to start at file_id=1 and parse every file to the last file ID listed in a latestfiles query. I haven't tried to parse out any of the new entries since idGames Archive has been re-enabled after the hardware swap.

Here's what I've found...

1) http://www.doomworld.com/idgames/api/api.php?action=get&id=1243&out=json

One of the rating objects in that record has unquoted value of >+1<; unquoted text is usually treated as a number, and >+1< is not a valid JSON number. A valid number with a plus sign would be a number with an exponent, so something like >1E+12345< is valid JSON.

2) http://www.doomworld.com/idgames/api/api.php?action=get&id=5159&out=json

The credits field has a unquoted value of >.4<, which is not a valid float value, it's missing the leading zero. It should either be quoted to make it text, or the leading zero added to make it a number. Since the credits field is usually text, I think you'll want to quote it.

3) http://www.doomworld.com/idgames/api/api.php?action=get&id=15306&out=json

One of the vote objects has a text field has an unquoted value of >0.<, which is not a valid float value. That value should be quoted, or made into a valid floating point number.

Maybe you could go through all of the fields in all of the API requests and decide beforehand if the field is a string or a number, and always apply quotes to fields that should always be strings in the JSON response?

Also, one other thing, the JSON >"meta":{"version":1.0}< is somewhat ambiguous; my parser is truncating the number and turning it into the integer 1. If the version ever gets incremented, are you expecting people who consume the API to compare versions as strings, or as a floating point numbers? If you're expecting people to compare versions as strings, then you need to quote the version to make JSON parsers parse the value as a string. If you're expecting people to compare floats, things could get real interesting when the last digit in your floating point version number is a >0<. Based on what I've seen so far, I'm expecting most parsers to truncate that last digit when it's a >0<. FWIW, the version value in the XML is quoted, so it's treated as a string.

JSON RFC for reference: http://tools.ietf.org/html/rfc4627

Share this post


Link to post

(Sent back to Spicyjack via E-mail, as well)

Whoa. Nice catches!

Part of the problem was I was relying on PHP's is_numeric() function for figuring out if something was numeric or not. While this solves for 99% of the cases, it is not strict enough for JSON, apparently.

Looks like I'll need to write something that's more strict.

Not sure what the best approach is for the version number. My intention was for the number to be interpreted as floating point. If most JSON parsers store their parsed data as JS-equivalent objects, it should still be interpreted as "Numeric, Floating Point". Hard to enforce that in XML, I'm afraid.

Thanks for your findings! I'll probably have it fixed this week.

Share this post


Link to post
rf` said:

I will marry the first person to code a search feature that allows you to search for names shorter than three letters. :V

Isn't that up to the server to solve?

Share this post


Link to post
MTrop said:

Part of the problem was I was relying on PHP's is_numeric() function for figuring out if something was numeric or not. While this solves for 99% of the cases, it is not strict enough for JSON, apparently.

Looks like I'll need to write something that's more strict.
[/B]/


The issues I found would make excellent test cases for whatever implementation you decide to work out ;)


Not sure what the best approach is for the version number. My intention was for the number to be interpreted as floating point. If most JSON parsers store their parsed data as JS-equivalent objects, it should still be interpreted as "Numeric, Floating Point". Hard to enforce that in XML, I'm afraid.


As floating point values, if your version is currently 1.9, and you bump to 1.10, the trailing 0 may^H^H^Hmost likely will get truncated, making the resulting version 1.1. Same thing when moving between 1.99 and 1.100. The extra trailing zeros at the end of a floating point number are unnecessary, and most likely would be stripped whenever the first math operation took place, depending on the implementation of the math library.

If the version was a string, then to the computer, all of the digits are important, and they would never be purposely stripped off.

No rush on the fixes, I'm currently using the XML for everything, this is more for people who write things in the future and want to consume the JSON.

Share this post


Link to post
rf` said:

I will marry the first person to code a search feature that allows you to search for names shorter than three letters. :V

printz said:

Isn't that up to the server to solve?


Only if you use the server for all searches. If you have a copy of the data, or even a subset of the data, then you can "solve" that problem however you want.

Share this post


Link to post
spicyjack said:

As floating point values, if your version is currently 1.9, and you bump to 1.10, the trailing 0 may^H^H^Hmost likely will get truncated, making the resulting version 1.1. Same thing when moving between 1.99 and 1.100. The extra trailing zeros at the end of a floating point number are unnecessary, and most likely would be stripped whenever the first math operation took place, depending on the implementation of the math library.

If the version was a string, then to the computer, all of the digits are important, and they would never be purposely stripped off.


Ooh, that's true. Maybe I should force output as a string, then.

Share this post


Link to post

Howdy, y'all.

Sorry for the delay, but I finally got around to fixing ALL numerics in JSON (I hope).

The only major thing that is changing is the output versions are now going to be integers, not floating-points.

Apologies if this causes problems in implementations.

If the API page displays "Updated: 1/2/2014", it has been applied.

EDIT/UPDATE:

Found a bug where "0." or "-" would evaluate as a numeric value (should be strings).

Fixed if API page displays "Updated: 1/2/2014 5:20PM".

Share this post


Link to post
MTrop said:

The only major thing that is changing is the output versions are now going to be integers, not floating-points.

Apologies if this causes problems in implementations.

If the API page displays "Updated: 1/2/2014", it has been applied.

EDIT/UPDATE:

Found a bug where "0." or "-" would evaluate as a numeric value (should be strings).

Fixed if API page displays "Updated: 1/2/2014 5:20PM".


I vote for an API version bump because of the change in API version attribute from a stringy-quasi-float to an integer. That way, no apologies would be required ;)

Also, a changelog of all of these changes somewhere on the API page itself would be awesome, so people who work with the API can match API versions to different functionality in the API. Maybe list all of the changes, major and minor, but major changes (where the behavior of the API changed) get highlighted in a different color?

I'll go through all of the files via JSON API requests again in the next few days to verify they can be parsed by my parser, and will update this thread when I'm done.

Share this post


Link to post
spicyjack said:

I'll go through all of the files via JSON API requests again in the next few days to verify they can be parsed by my parser, and will update this thread when I'm done.


I went through all of the API entries again, both in XML and JSON, and they both parsed out fine.

Thanks for all of the fixes.

Share this post


Link to post
×