Cyberdemon
Register | User Profile | Member List | F.A.Q | Privacy Policy | New Blog | Search Forums | Forums Home
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Misc. > Doomworld News > /idgames Archive Database REST API Now Available
Pages (2): « 1 [2]  
Author
All times are GMT. The time now is 23:53. Post New Thread    Post A Reply
MTrop
Member


Posts: 403
Registered: 12-05



spicyjack said:

I'll keep in mind the contact e-mail address mentioned above in the about call, however, any chance of posting the API script somewhere publicly, preferably some place with a issues/bug queue, like GitHub/Bitbucket/Sourceforge/etc.?

Thanks again!



No prob.

Unfortunately, the code will not be posted publicly, as Bloodshedder is a bit protective of code that would show off some of Doomworld's database layout, but feel free to E-mail me about any issues that may come up!

Old Post 09-22-13 07:32 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
rf`
Dr. Nosign


Posts: 8173
Registered: 11-01


I will marry the first person to code a search feature that allows you to search for names shorter than three letters. :V

Old Post 09-22-13 08:07 #
rf` is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Waffenak
Forum Regular


Posts: 720
Registered: 10-09


mobile apps rule

Old Post 10-12-13 20:06 #
Waffenak is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
spicyjack
Mini-Member


Posts: 56
Registered: 08-09


(Note: I also e-mailed this directly to MTrop, but posting it here so everyone can see it)

I've been parsing the contents of the idGames API over the last couple of weeks, and I found some more issues with the JSON. The XML requests are parsing fine, so no worries there.

My methodology was to start at file_id=1 and parse every file to the last file ID listed in a latestfiles query. I haven't tried to parse out any of the new entries since idGames Archive has been re-enabled after the hardware swap.

Here's what I've found...

1) http://www.doomworld.com/idgames/ap...d=1243&out=json

One of the rating objects in that record has unquoted value of >+1<; unquoted text is usually treated as a number, and >+1< is not a valid JSON number. A valid number with a plus sign would be a number with an exponent, so something like >1E+12345< is valid JSON.

2) http://www.doomworld.com/idgames/ap...d=5159&out=json

The credits field has a unquoted value of >.4<, which is not a valid float value, it's missing the leading zero. It should either be quoted to make it text, or the leading zero added to make it a number. Since the credits field is usually text, I think you'll want to quote it.

3) http://www.doomworld.com/idgames/ap...=15306&out=json

One of the vote objects has a text field has an unquoted value of >0.<, which is not a valid float value. That value should be quoted, or made into a valid floating point number.

Maybe you could go through all of the fields in all of the API requests and decide beforehand if the field is a string or a number, and always apply quotes to fields that should always be strings in the JSON response?

Also, one other thing, the JSON >"meta":{"version":1.0}< is somewhat ambiguous; my parser is truncating the number and turning it into the integer 1. If the version ever gets incremented, are you expecting people who consume the API to compare versions as strings, or as a floating point numbers? If you're expecting people to compare versions as strings, then you need to quote the version to make JSON parsers parse the value as a string. If you're expecting people to compare floats, things could get real interesting when the last digit in your floating point version number is a >0<. Based on what I've seen so far, I'm expecting most parsers to truncate that last digit when it's a >0<. FWIW, the version value in the XML is quoted, so it's treated as a string.

JSON RFC for reference: http://tools.ietf.org/html/rfc4627

Old Post 12-22-13 23:44 #
spicyjack is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 403
Registered: 12-05


(Sent back to Spicyjack via E-mail, as well)

Whoa. Nice catches!

Part of the problem was I was relying on PHP's is_numeric() function for figuring out if something was numeric or not. While this solves for 99% of the cases, it is not strict enough for JSON, apparently.

Looks like I'll need to write something that's more strict.

Not sure what the best approach is for the version number. My intention was for the number to be interpreted as floating point. If most JSON parsers store their parsed data as JS-equivalent objects, it should still be interpreted as "Numeric, Floating Point". Hard to enforce that in XML, I'm afraid.

Thanks for your findings! I'll probably have it fixed this week.

Old Post 12-23-13 20:03 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
printz
CRAZY DUMB ZEALOT


Posts: 8552
Registered: 06-06



rf` said:
I will marry the first person to code a search feature that allows you to search for names shorter than three letters. :V
Isn't that up to the server to solve?

__________________
Automatic Wolfenstein - Version 1.0

Old Post 12-23-13 21:16 #
printz is online now Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
spicyjack
Mini-Member


Posts: 56
Registered: 08-09



MTrop said:

Part of the problem was I was relying on PHP's is_numeric() function for figuring out if something was numeric or not. While this solves for 99% of the cases, it is not strict enough for JSON, apparently.

Looks like I'll need to write something that's more strict.
/


The issues I found would make excellent test cases for whatever implementation you decide to work out ;)



Not sure what the best approach is for the version number. My intention was for the number to be interpreted as floating point. If most JSON parsers store their parsed data as JS-equivalent objects, it should still be interpreted as "Numeric, Floating Point". Hard to enforce that in XML, I'm afraid.



As floating point values, if your version is currently 1.9, and you bump to 1.10, the trailing 0 may^H^H^Hmost likely will get truncated, making the resulting version 1.1. Same thing when moving between 1.99 and 1.100. The extra trailing zeros at the end of a floating point number are unnecessary, and most likely would be stripped whenever the first math operation took place, depending on the implementation of the math library.

If the version was a string, then to the computer, all of the digits are important, and they would never be purposely stripped off.

No rush on the fixes, I'm currently using the XML for everything, this is more for people who write things in the future and want to consume the JSON.

Old Post 12-23-13 21:36 #
spicyjack is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
spicyjack
Mini-Member


Posts: 56
Registered: 08-09



rf` said:
I will marry the first person to code a search feature that allows you to search for names shorter than three letters. :V



printz said:
Isn't that up to the server to solve?


Only if you use the server for all searches. If you have a copy of the data, or even a subset of the data, then you can "solve" that problem however you want.

Old Post 12-23-13 21:45 #
spicyjack is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 403
Registered: 12-05



spicyjack said:

As floating point values, if your version is currently 1.9, and you bump to 1.10, the trailing 0 may^H^H^Hmost likely will get truncated, making the resulting version 1.1. Same thing when moving between 1.99 and 1.100. The extra trailing zeros at the end of a floating point number are unnecessary, and most likely would be stripped whenever the first math operation took place, depending on the implementation of the math library.

If the version was a string, then to the computer, all of the digits are important, and they would never be purposely stripped off.



Ooh, that's true. Maybe I should force output as a string, then.

Old Post 12-24-13 08:09 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 403
Registered: 12-05


Howdy, y'all.

Sorry for the delay, but I finally got around to fixing ALL numerics in JSON (I hope).

The only major thing that is changing is the output versions are now going to be integers, not floating-points.

Apologies if this causes problems in implementations.

If the API page displays "Updated: 1/2/2014", it has been applied.

EDIT/UPDATE:

Found a bug where "0." or "-" would evaluate as a numeric value (should be strings).

Fixed if API page displays "Updated: 1/2/2014 5:20PM".

Last edited by MTrop on 01-02-14 at 22:21

Old Post 01-02-14 07:45 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
spicyjack
Mini-Member


Posts: 56
Registered: 08-09



MTrop said:

The only major thing that is changing is the output versions are now going to be integers, not floating-points.

Apologies if this causes problems in implementations.

If the API page displays "Updated: 1/2/2014", it has been applied.

EDIT/UPDATE:

Found a bug where "0." or "-" would evaluate as a numeric value (should be strings).

Fixed if API page displays "Updated: 1/2/2014 5:20PM".



I vote for an API version bump because of the change in API version attribute from a stringy-quasi-float to an integer. That way, no apologies would be required ;)

Also, a changelog of all of these changes somewhere on the API page itself would be awesome, so people who work with the API can match API versions to different functionality in the API. Maybe list all of the changes, major and minor, but major changes (where the behavior of the API changed) get highlighted in a different color?

I'll go through all of the files via JSON API requests again in the next few days to verify they can be parsed by my parser, and will update this thread when I'm done.

Old Post 01-04-14 20:20 #
spicyjack is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
spicyjack
Mini-Member


Posts: 56
Registered: 08-09



spicyjack said:

I'll go through all of the files via JSON API requests again in the next few days to verify they can be parsed by my parser, and will update this thread when I'm done.



I went through all of the API entries again, both in XML and JSON, and they both parsed out fine.

Thanks for all of the fixes.

Old Post 01-15-14 04:14 #
spicyjack is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
All times are GMT. The time now is 23:53. Post New Thread    Post A Reply
Pages (2): « 1 [2]  
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Misc. > Doomworld News > /idgames Archive Database REST API Now Available

Show Printable Version | Email this Page | Subscribe to this Thread

 

Forum Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is OFF
vB code is ON
Smilies are OFF
[IMG] code is OFF
 

< Contact Us - Doomworld >

Powered by: vBulletin Version 2.2.5
Copyright ©2000, 2001, Jelsoft Enterprises Limited.

Message Board Statistics