/idgames Archive Database REST API Now Available

Wouldn't hurt, since the /idgames page works poorly in mobile browsers.

Share this post


Link to post

This API looks fantastic. Nice work.

However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

Again, fantastic work...it's just probably not a good idea to call it "REST".

Share this post


Link to post

Thanks for going out of your way to do this, what a great idea.

Share this post


Link to post

This is very awesome! A great idea - it's similar to something I've been working on at home, to organize my own archives :)
I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

Once again, very nice - good job!

Share this post


Link to post
AlexMax said:

This API looks fantastic. Nice work.

However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

Again, fantastic work...it's just probably not a good idea to call it "REST".


Yeah, you're right. It's not SOAP, and it isn't quite REST, but it's stateless, and some kind of in-the-middle thing. Maybe I'll just call it, what, a "Public API?" Get rid of the specific model type?

kb1 said:

I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

Once again, very nice - good job!


If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".

Share this post


Link to post
Mista_T said:

If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".

I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.

Share this post


Link to post
kb1 said:

I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.


You DO get the ID number from the XML. latestfiles with a limit of 1 will get you that.

Would you also like a range? Like from an id number?

Share this post


Link to post

Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

Share this post


Link to post
DaniJ said:

Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

I concur.

Share this post


Link to post
DaniJ said:

Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.


Seconded. In writing the Android app I spent far too much time writing an XML parser that works at any decent speed. I don't know if using Android's built-in JSON parser is going to be much faster, but I wouldn't have to maintain 12 Kb of mostly if-elseif code for it.

Share this post


Link to post

JSON output was something I was planning on doing. Now that there's demand, it will probably be the next thing I do. I hope that PHP (and Doomworld's PHP version) have something that will make this simpler. If not, though, it wouldn't be too hard to write.

Share this post


Link to post

Doomworld is running PHP 5.3.10. I would be very surprised if it didn't have JSON support compiled in.

Share this post


Link to post
AlexMax said:

Doomworld is running PHP 5.3.10. I would be very surprised if it didn't have JSON support compiled in.


Yeah, but it didn't have some recent DB connection libraries compiled into it so I'm not holding out hope...

EDIT: "Recent" meaning the last couple of years. YEARS.

Share this post


Link to post

Houston, we have JSON. I repeat, we have JSON.

Also, that little "latestfiles" extension too. Feed it a "startid" parameter in order to tell it what id to start at if you don't want it to start at the latest one.

Share this post


Link to post
Mista_T said:

Houston, we have JSON. I repeat, we have JSON.

Johnson and Johnson.

Share this post


Link to post
Mista_T said:

Houston, we have JSON. I repeat, we have JSON.

Excellent. I'll start prototyping some of the applications I have planned and let you know if I encounter any use cases which I feel aren't adequately served by the API. Though my initial thoughts are that it looks pretty good.

Share this post


Link to post

I've just noticed that there may be some instances where a PHP array of one object may be returned as an Object and not an Array with one Object in it in JSON, so that may require a tiny bit of extra coding should you choose to parse JSON in JavaScript or something else. I really, really wished that PHP had separated Arrays and Associated Arrays into two distinct types...

I'm kinda afraid to mess with it because that would mean I could end up breaking structure for one output format versus the other, so I'm holding off unless it becomes a greater issue.

Share this post


Link to post

Some suggestions / bug reports



Insufficient backslash escaping in json output

Consider the description field returned when querying eaeuro03 (id=11518)

"description":"A fairly detailed map for Doom2. Inspired by
Fredik Johansson's incredible Vrack series. This map may cause a lot
of problems on slower computers. If your computer is slower than PII
200 Mhz, don't even try to play this level, sorry :",
Notice how the smiley at the end escapes the closing quote? This breaks both json parsers that come with perl 5.14 and python 3.2.



not all fields are always defined

The documentation claims that several fields in returned search queries are always present when they are not. Something like comsw (id=12) or any file lacking an accompanying .txt (e.g. everything in themes/DAC/dac2005) have none of these fields.



index.php in url (cosmetic)

If it is easy please consider not appending "index.php" in returned urls.
http://www.doomworld.com/idgames/index.php?id=
can be shortened to
http://www.doomworld.com/idgames/?id=
which looks neater. index.php is just noise really.

Share this post


Link to post
RjY said:

Some suggestions / bug reports[/b]

Insufficient backslash escaping in json output


Fixed. Came about because preg_replace was being weird about escaping backslashes in regex for some weird reason. Was easier to do str_replace instead.

[b]not all fields are always defined


Done-ish.

"Blank" fields will be "null" valued or whatever the DB is returning. I keep forgetting that undefined != null in JavaScript. Of course, as always, what is available is what the idGames parser picks up from the TXT when it is added to the archive. Disclaimer added to "search" function, just like "get" and "getfiles".

index.php in url (cosmetic)


Done.


Told Bloodshedder about new fixes. If you see "Updated: 12/21/2012" on the API page under the title, it's been applied.

EDIT: Found another string escaping bug that needed fixing. Should be fixed soon.

Share this post


Link to post

I noticed that the /idgames site now displays ratings and usernames for individual reviews. Would it be possible for the API to include these details so I can add them to the Android app?

Share this post


Link to post
MTrop said:

Houston, we have JSON. I repeat, we have JSON.

I have a bug to report.

For this entry in idgames:

http://www.doomworld.com/idgames/index.php?id=17259

The author entered in some characters using ISO-8859-1; notice the 'small letter a acute' and 'small letter o acute' in the author's name. The hexadecimal bytes for these letters are 0xe1 and 0xf3 respectively.

In the XML API call [1], it converts these bytes to the proper UTF-8 byte sequences (0xc3 0xa1 for the acute 'a' and 0xc3 0xb3 for the acute 'o').

curl "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259" \
 | xxd - | less
...
00000f0: 332d 3036 2d32 393c 2f64 6174 653e 3c61  3-06-29</date><a
0000100: 7574 686f 723e 5a6f 6c74 c3a1 6e20 53c3  uthor>Zolt..n S.
0000110: b366 616c 7669 2028 5a38 3629 3c2f 6175  .falvi (Z86)</au
0000120: 7468 6f72 3e3c 656d 6169 6c3e 7a65 7261  thor><email>zera
But the JSON [2] is not converted to valid UTF-8, the bytes remain encoded in ISO-8859-1, and those bytes are invalid sequences in UTF-8, so parsers most likely won't parse it. Valid JSON per RFC4627 is Unicode, so UTF-8, UTF-16 or UTF-32 (http://www.ietf.org/rfc/rfc4627.txt).
curl "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json" \
  | xxd - | less
...
0000090: 3a22 3230 3133 2d30 362d 3239 222c 2261  :"2013-06-29","a
00000a0: 7574 686f 7222 3a22 5a6f 6c74 e16e 2053  uthor":"Zolt.n S
00000b0: f366 616c 7669 2028 5a38 3629 222c 2265  .falvi (Z86)","e
The parser I was using was unhappy until I manually scanned for the offending bytes and converted to UTF-8 prior to unserializing the JSON into objects.

I can switch to using the XML version of the API, but I would prefer to use the JSON version if possible.

For what it's worth, I found a few decent pages [3][4] showing byte encodings for UTF-8, if anyone needs them for future reference;

[1] http://www.doomworld.com/idgames/api/api.php?action=get&id=17259
[2] http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json
[3] http://www.utf8-chartable.de/
[4] http://dev.networkerror.org/utf8

[edit] added valid encoding types for JSON per the RFC and a link to the RFC[/edit]

Share this post


Link to post

Whoa. Lucky for you I decided to check up on this thread!

I'll take a look. Shouldn't be too terrible. The conversion is happening for the XML version, but I guess that I forgot about it for JSON.

If you need to get in touch, I'm reachable by the E-mail address in the ABOUT call.

Share this post


Link to post

Okay. I've sent off the change to bloodshedder.

If you see the date at the top of the API page changed to 9/19/2013, it's been applied.

EDIT: Okay - looks like we're good here. Not sure if PHP is sending back the correct encoding headers, but according to Fiddler2, it's readable as UTF-8. Are you able to double-check?

Share this post


Link to post
MTrop said:

Okay. I've sent off the change to bloodshedder.

If you see the date at the top of the API page changed to 9/19/2013, it's been applied.

EDIT: Okay - looks like we're good here. Not sure if PHP is sending back the correct encoding headers, but according to Fiddler2, it's readable as UTF-8. Are you able to double-check?


Cool, thanks for the super-quick turnaround, I see the updated script live now.

Here's what I get now:

wget -O - "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json" | xxd | less
{snip}
00000a0: 7574 686f 7222 3a22 5a6f 6c74 c3a1 6e20  uthor":"Zolt..n 
00000b0: 53c3 b366 616c 7669 2028 5a38 3629 222c  S..falvi (Z86)",
So the 0xe1 is now 0xc3 0xa1, and 0xf3 is now 0xc3 0xb3, both of which are the valid UTF-8 replacement sequences for the offending ISO-8859-1 characters (according to http://www.utf8-chartable.de/).

This was actually a good lesson for me, it reminded me that I ALWAYS need to validate my input from external sources :) If I find anything else wonky, I'll be sure to let you know.

As far as the headers:
$ wget --spider --server-response "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json"
Spider mode enabled. Check if remote file exists.
--2013-09-21 11:17:25--  http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json
Resolving doomworld.com... 38.68.5.148
Connecting to doomworld.com|38.68.5.148|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Sat, 21 Sep 2013 18:17:22 GMT
  Server: Apache/2.2.23 (Unix) PHP/5.4.10
  X-Powered-By: PHP/5.4.10
  Keep-Alive: timeout=15, max=500
  Connection: Keep-Alive
  Content-Type: application/json
Length: unspecified [application/json]
No, no encoding type mentioned in the headers. For what it's worth, the XML output also doesn't have encoding type in the headers, but since character encoding is part of the document, it's not such a big deal.

I'll keep in mind the contact e-mail address mentioned above in the about call, however, any chance of posting the API script somewhere publicly, preferably some place with a issues/bug queue, like GitHub/Bitbucket/Sourceforge/etc.?

Thanks again!

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now