Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
  • Sign in to follow this  

    /idgames Archive Database REST API Now Available


    MTrop

    The /idgames Archive database now has a public-facing REST API which will enable programmers to write applications that can interface with it. The documentation for the API is available here.

    Sign in to follow this  


    User Feedback

    Recommended Comments



    This API looks fantastic. Nice work.

    However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

    Again, fantastic work...it's just probably not a good idea to call it "REST".

    Share this comment


    Link to comment

    This is very awesome! A great idea - it's similar to something I've been working on at home, to organize my own archives :)
    I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

    Once again, very nice - good job!

    Share this comment


    Link to comment
    AlexMax said:

    This API looks fantastic. Nice work.

    However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

    Again, fantastic work...it's just probably not a good idea to call it "REST".


    Yeah, you're right. It's not SOAP, and it isn't quite REST, but it's stateless, and some kind of in-the-middle thing. Maybe I'll just call it, what, a "Public API?" Get rid of the specific model type?

    kb1 said:

    I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

    Once again, very nice - good job!


    If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".

    Share this comment


    Link to comment
    Mista_T said:

    If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".

    I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

    It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.

    Share this comment


    Link to comment
    kb1 said:

    I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

    It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.


    You DO get the ID number from the XML. latestfiles with a limit of 1 will get you that.

    Would you also like a range? Like from an id number?

    Share this comment


    Link to comment

    Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

    Share this comment


    Link to comment
    DaniJ said:

    Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

    I concur.

    Share this comment


    Link to comment
    DaniJ said:

    Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.


    Seconded. In writing the Android app I spent far too much time writing an XML parser that works at any decent speed. I don't know if using Android's built-in JSON parser is going to be much faster, but I wouldn't have to maintain 12 Kb of mostly if-elseif code for it.

    Share this comment


    Link to comment

    JSON output was something I was planning on doing. Now that there's demand, it will probably be the next thing I do. I hope that PHP (and Doomworld's PHP version) have something that will make this simpler. If not, though, it wouldn't be too hard to write.

    Share this comment


    Link to comment
    AlexMax said:

    Doomworld is running PHP 5.3.10. I would be very surprised if it didn't have JSON support compiled in.


    Yeah, but it didn't have some recent DB connection libraries compiled into it so I'm not holding out hope...

    EDIT: "Recent" meaning the last couple of years. YEARS.

    Share this comment


    Link to comment

    Houston, we have JSON. I repeat, we have JSON.

    Also, that little "latestfiles" extension too. Feed it a "startid" parameter in order to tell it what id to start at if you don't want it to start at the latest one.

    Share this comment


    Link to comment
    Mista_T said:

    Houston, we have JSON. I repeat, we have JSON.

    Excellent. I'll start prototyping some of the applications I have planned and let you know if I encounter any use cases which I feel aren't adequately served by the API. Though my initial thoughts are that it looks pretty good.

    Share this comment


    Link to comment

    I've just noticed that there may be some instances where a PHP array of one object may be returned as an Object and not an Array with one Object in it in JSON, so that may require a tiny bit of extra coding should you choose to parse JSON in JavaScript or something else. I really, really wished that PHP had separated Arrays and Associated Arrays into two distinct types...

    I'm kinda afraid to mess with it because that would mean I could end up breaking structure for one output format versus the other, so I'm holding off unless it becomes a greater issue.

    Share this comment


    Link to comment

    Some suggestions / bug reports



    Insufficient backslash escaping in json output

    Consider the description field returned when querying eaeuro03 (id=11518)

    "description":"A fairly detailed map for Doom2. Inspired by
    Fredik Johansson's incredible Vrack series. This map may cause a lot
    of problems on slower computers. If your computer is slower than PII
    200 Mhz, don't even try to play this level, sorry :",
    Notice how the smiley at the end escapes the closing quote? This breaks both json parsers that come with perl 5.14 and python 3.2.



    not all fields are always defined

    The documentation claims that several fields in returned search queries are always present when they are not. Something like comsw (id=12) or any file lacking an accompanying .txt (e.g. everything in themes/DAC/dac2005) have none of these fields.



    index.php in url (cosmetic)

    If it is easy please consider not appending "index.php" in returned urls.
    http://www.doomworld.com/idgames/index.php?id=
    can be shortened to
    http://www.doomworld.com/idgames/?id=
    which looks neater. index.php is just noise really.

    Share this comment


    Link to comment
    RjY said:

    Some suggestions / bug reports[/b]

    Insufficient backslash escaping in json output


    Fixed. Came about because preg_replace was being weird about escaping backslashes in regex for some weird reason. Was easier to do str_replace instead.

    [b]not all fields are always defined


    Done-ish.

    "Blank" fields will be "null" valued or whatever the DB is returning. I keep forgetting that undefined != null in JavaScript. Of course, as always, what is available is what the idGames parser picks up from the TXT when it is added to the archive. Disclaimer added to "search" function, just like "get" and "getfiles".

    index.php in url (cosmetic)


    Done.


    Told Bloodshedder about new fixes. If you see "Updated: 12/21/2012" on the API page under the title, it's been applied.

    EDIT: Found another string escaping bug that needed fixing. Should be fixed soon.

    Share this comment


    Link to comment

    I noticed that the /idgames site now displays ratings and usernames for individual reviews. Would it be possible for the API to include these details so I can add them to the Android app?

    Share this comment


    Link to comment
    MTrop said:

    Houston, we have JSON. I repeat, we have JSON.

    I have a bug to report.

    For this entry in idgames:

    http://www.doomworld.com/idgames/index.php?id=17259

    The author entered in some characters using ISO-8859-1; notice the 'small letter a acute' and 'small letter o acute' in the author's name. The hexadecimal bytes for these letters are 0xe1 and 0xf3 respectively.

    In the XML API call [1], it converts these bytes to the proper UTF-8 byte sequences (0xc3 0xa1 for the acute 'a' and 0xc3 0xb3 for the acute 'o').

    curl "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259" \
     | xxd - | less
    ...
    00000f0: 332d 3036 2d32 393c 2f64 6174 653e 3c61  3-06-29</date><a
    0000100: 7574 686f 723e 5a6f 6c74 c3a1 6e20 53c3  uthor>Zolt..n S.
    0000110: b366 616c 7669 2028 5a38 3629 3c2f 6175  .falvi (Z86)</au
    0000120: 7468 6f72 3e3c 656d 6169 6c3e 7a65 7261  thor><email>zera
    
    But the JSON [2] is not converted to valid UTF-8, the bytes remain encoded in ISO-8859-1, and those bytes are invalid sequences in UTF-8, so parsers most likely won't parse it. Valid JSON per RFC4627 is Unicode, so UTF-8, UTF-16 or UTF-32 (http://www.ietf.org/rfc/rfc4627.txt).
    curl "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json" \
      | xxd - | less
    ...
    0000090: 3a22 3230 3133 2d30 362d 3239 222c 2261  :"2013-06-29","a
    00000a0: 7574 686f 7222 3a22 5a6f 6c74 e16e 2053  uthor":"Zolt.n S
    00000b0: f366 616c 7669 2028 5a38 3629 222c 2265  .falvi (Z86)","e
    
    The parser I was using was unhappy until I manually scanned for the offending bytes and converted to UTF-8 prior to unserializing the JSON into objects.

    I can switch to using the XML version of the API, but I would prefer to use the JSON version if possible.

    For what it's worth, I found a few decent pages [3][4] showing byte encodings for UTF-8, if anyone needs them for future reference;

    [1] http://www.doomworld.com/idgames/api/api.php?action=get&id=17259
    [2] http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json
    [3] http://www.utf8-chartable.de/
    [4] http://dev.networkerror.org/utf8

    [edit] added valid encoding types for JSON per the RFC and a link to the RFC[/edit]

    Share this comment


    Link to comment

    Whoa. Lucky for you I decided to check up on this thread!

    I'll take a look. Shouldn't be too terrible. The conversion is happening for the XML version, but I guess that I forgot about it for JSON.

    If you need to get in touch, I'm reachable by the E-mail address in the ABOUT call.

    Share this comment


    Link to comment

    Okay. I've sent off the change to bloodshedder.

    If you see the date at the top of the API page changed to 9/19/2013, it's been applied.

    EDIT: Okay - looks like we're good here. Not sure if PHP is sending back the correct encoding headers, but according to Fiddler2, it's readable as UTF-8. Are you able to double-check?

    Share this comment


    Link to comment
    MTrop said:

    Okay. I've sent off the change to bloodshedder.

    If you see the date at the top of the API page changed to 9/19/2013, it's been applied.

    EDIT: Okay - looks like we're good here. Not sure if PHP is sending back the correct encoding headers, but according to Fiddler2, it's readable as UTF-8. Are you able to double-check?


    Cool, thanks for the super-quick turnaround, I see the updated script live now.

    Here's what I get now:

    wget -O - "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json" | xxd | less
    {snip}
    00000a0: 7574 686f 7222 3a22 5a6f 6c74 c3a1 6e20  uthor":"Zolt..n 
    00000b0: 53c3 b366 616c 7669 2028 5a38 3629 222c  S..falvi (Z86)",
    
    So the 0xe1 is now 0xc3 0xa1, and 0xf3 is now 0xc3 0xb3, both of which are the valid UTF-8 replacement sequences for the offending ISO-8859-1 characters (according to http://www.utf8-chartable.de/).

    This was actually a good lesson for me, it reminded me that I ALWAYS need to validate my input from external sources :) If I find anything else wonky, I'll be sure to let you know.

    As far as the headers:
    $ wget --spider --server-response "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json"
    Spider mode enabled. Check if remote file exists.
    --2013-09-21 11:17:25--  http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json
    Resolving doomworld.com... 38.68.5.148
    Connecting to doomworld.com|38.68.5.148|:80... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 200 OK
      Date: Sat, 21 Sep 2013 18:17:22 GMT
      Server: Apache/2.2.23 (Unix) PHP/5.4.10
      X-Powered-By: PHP/5.4.10
      Keep-Alive: timeout=15, max=500
      Connection: Keep-Alive
      Content-Type: application/json
    Length: unspecified [application/json]
    
    No, no encoding type mentioned in the headers. For what it's worth, the XML output also doesn't have encoding type in the headers, but since character encoding is part of the document, it's not such a big deal.

    I'll keep in mind the contact e-mail address mentioned above in the about call, however, any chance of posting the API script somewhere publicly, preferably some place with a issues/bug queue, like GitHub/Bitbucket/Sourceforge/etc.?

    Thanks again!

    Share this comment


    Link to comment
    spicyjack said:

    I'll keep in mind the contact e-mail address mentioned above in the about call, however, any chance of posting the API script somewhere publicly, preferably some place with a issues/bug queue, like GitHub/Bitbucket/Sourceforge/etc.?

    Thanks again!


    No prob.

    Unfortunately, the code will not be posted publicly, as Bloodshedder is a bit protective of code that would show off some of Doomworld's database layout, but feel free to E-mail me about any issues that may come up!

    Share this comment


    Link to comment



    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now

×