Ouchface
Register | User Profile | Member List | F.A.Q | Privacy Policy | New Blog | Search Forums | Forums Home
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Misc. > Doomworld News > /idgames Archive Database REST API Now Available
Pages (2): [1] 2 »  
Author
All times are GMT. The time now is 09:34. Post New Thread    Post A Reply
MTrop
Member


Posts: 421
Registered: 12-05


The /idgames Archive database now has a public-facing REST API which will enable programmers to write applications that can interface with it. The documentation for the API is available here.

Old Post 12-14-12 16:57 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
GhostlyDeath
Forum Retard


Posts: 1027
Registered: 08-05


Neat, i'll have to code something up for this.

Old Post 12-14-12 20:20 #
GhostlyDeath is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
printz
CRAZY DUMB ZEALOT


Posts: 8871
Registered: 06-06


Does this mean there'll be mobile apps for /idgames? :)

__________________
Automatic Wolfenstein - Version 1.0 - also on Android

Old Post 12-14-12 21:15 #
printz is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
kristus
Megablast!


Posts: 10608
Registered: 07-00


Wouldn't hurt, since the /idgames page works poorly in mobile browsers.

Old Post 12-14-12 21:36 #
kristus is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05



printz said:
Does this mean there'll be mobile apps for /idgames? :)


exl already made this app for Android Phones. It's pretty spiffy.

Old Post 12-14-12 21:36 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
AlexMax
Senior Member


Posts: 1114
Registered: 01-03


This API looks fantastic. Nice work.

However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

Again, fantastic work...it's just probably not a good idea to call it "REST".

Old Post 12-14-12 23:39 #
AlexMax is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
Use3D
Forum Legend


Posts: 4714
Registered: 04-02


Thanks for going out of your way to do this, what a great idea.

Old Post 12-14-12 23:48 #
Use3D is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
kb1
Member


Posts: 362
Registered: 11-06


This is very awesome! A great idea - it's similar to something I've been working on at home, to organize my own archives :)
I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

Once again, very nice - good job!

Old Post 12-15-12 01:11 #
kb1 is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05



AlexMax said:
This API looks fantastic. Nice work.

However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

Again, fantastic work...it's just probably not a good idea to call it "REST".



Yeah, you're right. It's not SOAP, and it isn't quite REST, but it's stateless, and some kind of in-the-middle thing. Maybe I'll just call it, what, a "Public API?" Get rid of the specific model type?


kb1 said:
I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

Once again, very nice - good job!



If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".

Last edited by MTrop on 12-15-12 at 01:17

Old Post 12-15-12 01:12 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
kb1
Member


Posts: 362
Registered: 11-06



Mista_T said:If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".
I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.

Old Post 12-15-12 02:19 #
kb1 is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05



kb1 said:
I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.



You DO get the ID number from the XML. latestfiles with a limit of 1 will get you that.

Would you also like a range? Like from an id number?

Old Post 12-15-12 03:01 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
DaniJ
Senior Member


Posts: 2120
Registered: 08-03


Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

Old Post 12-15-12 09:33 #
DaniJ is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
boris
meow


Posts: 3629
Registered: 05-00



DaniJ said:
Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

I concur.

Old Post 12-15-12 10:30 #
boris is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
exl
Personal Test Dummy


Posts: 578
Registered: 05-02



DaniJ said:
Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.


Seconded. In writing the Android app I spent far too much time writing an XML parser that works at any decent speed. I don't know if using Android's built-in JSON parser is going to be much faster, but I wouldn't have to maintain 12 Kb of mostly if-elseif code for it.

Old Post 12-15-12 10:59 #
exl is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
fraggle
Filled with the code of Doom


Posts: 7665
Registered: 07-00


Make this a fourth vote for a JSON option.

Old Post 12-15-12 16:13 #
fraggle is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
Graf Zahl
Why don't I have a custom title by now?!


Posts: 7747
Registered: 01-03


JSON fifthed.

Old Post 12-15-12 16:19 #
Graf Zahl is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05


JSON output was something I was planning on doing. Now that there's demand, it will probably be the next thing I do. I hope that PHP (and Doomworld's PHP version) have something that will make this simpler. If not, though, it wouldn't be too hard to write.

Last edited by MTrop on 12-15-12 at 16:40

Old Post 12-15-12 16:24 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
AlexMax
Senior Member


Posts: 1114
Registered: 01-03


Doomworld is running PHP 5.3.10. I would be very surprised if it didn't have JSON support compiled in.

Old Post 12-16-12 00:13 #
AlexMax is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05



AlexMax said:
Doomworld is running PHP 5.3.10. I would be very surprised if it didn't have JSON support compiled in.


Yeah, but it didn't have some recent DB connection libraries compiled into it so I'm not holding out hope...

EDIT: "Recent" meaning the last couple of years. YEARS.

Old Post 12-16-12 00:31 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05


Houston, we have JSON. I repeat, we have JSON.

Also, that little "latestfiles" extension too. Feed it a "startid" parameter in order to tell it what id to start at if you don't want it to start at the latest one.

Old Post 12-16-12 07:44 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
printz
CRAZY DUMB ZEALOT


Posts: 8871
Registered: 06-06



Mista_T said:
Houston, we have JSON. I repeat, we have JSON.
Johnson and Johnson.

__________________
Automatic Wolfenstein - Version 1.0 - also on Android

Old Post 12-16-12 07:55 #
printz is offline Profile || Blog || PM || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
DaniJ
Senior Member


Posts: 2120
Registered: 08-03



Mista_T said:
Houston, we have JSON. I repeat, we have JSON.

Excellent. I'll start prototyping some of the applications I have planned and let you know if I encounter any use cases which I feel aren't adequately served by the API. Though my initial thoughts are that it looks pretty good.

Old Post 12-16-12 10:11 #
DaniJ is offline Profile || Blog || PM || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05


I've just noticed that there may be some instances where a PHP array of one object may be returned as an Object and not an Array with one Object in it in JSON, so that may require a tiny bit of extra coding should you choose to parse JSON in JavaScript or something else. I really, really wished that PHP had separated Arrays and Associated Arrays into two distinct types...

I'm kinda afraid to mess with it because that would mean I could end up breaking structure for one output format versus the other, so I'm holding off unless it becomes a greater issue.

Old Post 12-17-12 01:21 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
RjY
anARCHy


Posts: 959
Registered: 05-02


Some suggestions / bug reports



Insufficient backslash escaping in json output

Consider the description field returned when querying eaeuro03 (id=11518)
code:
"description":"A fairly detailed map for Doom2. Inspired by Fredik Johansson's incredible Vrack series. This map may cause a lot of problems on slower computers. If your computer is slower than PII 200 Mhz, don't even try to play this level, sorry :\",

Notice how the smiley at the end escapes the closing quote? This breaks both json parsers that come with perl 5.14 and python 3.2.



not all fields are always defined

The documentation claims that several fields in returned search queries are always present when they are not. Something like comsw (id=12) or any file lacking an accompanying .txt (e.g. everything in themes/DAC/dac2005) have none of these fields.



index.php in url (cosmetic)

If it is easy please consider not appending "index.php" in returned urls.
code:
http://www.doomworld.com/idgames/index.php?id=
can be shortened to
code:
http://www.doomworld.com/idgames/?id=
which looks neater. index.php is just noise really.

Old Post 12-20-12 21:10 #
RjY is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05



RjY said:
Some suggestions / bug reports

Insufficient backslash escaping in json output



Fixed. Came about because preg_replace was being weird about escaping backslashes in regex for some weird reason. Was easier to do str_replace instead.



not all fields are always defined



Done-ish.

"Blank" fields will be "null" valued or whatever the DB is returning. I keep forgetting that undefined != null in JavaScript. Of course, as always, what is available is what the idGames parser picks up from the TXT when it is added to the archive. Disclaimer added to "search" function, just like "get" and "getfiles".



index.php in url (cosmetic)



Done.


Told Bloodshedder about new fixes. If you see "Updated: 12/21/2012" on the API page under the title, it's been applied.

EDIT: Found another string escaping bug that needed fixing. Should be fixed soon.

Last edited by MTrop on 12-21-12 at 19:08

Old Post 12-21-12 07:32 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
exl
Personal Test Dummy


Posts: 578
Registered: 05-02


I noticed that the /idgames site now displays ratings and usernames for individual reviews. Would it be possible for the API to include these details so I can add them to the Android app?

Old Post 08-22-13 18:29 #
exl is offline Profile || Blog || PM || Email || Search || Add Buddy IP || Edit/Delete || Quote
spicyjack
Mini-Member


Posts: 61
Registered: 08-09



MTrop said:
Houston, we have JSON. I repeat, we have JSON.


I have a bug to report.

For this entry in idgames:

http://www.doomworld.com/idgames/index.php?id=17259

The author entered in some characters using ISO-8859-1; notice the 'small letter a acute' and 'small letter o acute' in the author's name. The hexadecimal bytes for these letters are 0xe1 and 0xf3 respectively.

In the XML API call [1], it converts these bytes to the proper UTF-8 byte sequences (0xc3 0xa1 for the acute 'a' and 0xc3 0xb3 for the acute 'o').

code:
curl "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259" \ | xxd - | less ... 00000f0: 332d 3036 2d32 393c 2f64 6174 653e 3c61 3-06-29</date><a 0000100: 7574 686f 723e 5a6f 6c74 c3a1 6e20 53c3 uthor>Zolt..n S. 0000110: b366 616c 7669 2028 5a38 3629 3c2f 6175 .falvi (Z86)</au 0000120: 7468 6f72 3e3c 656d 6169 6c3e 7a65 7261 thor><email>zera


But the JSON [2] is not converted to valid UTF-8, the bytes remain encoded in ISO-8859-1, and those bytes are invalid sequences in UTF-8, so parsers most likely won't parse it. Valid JSON per RFC4627 is Unicode, so UTF-8, UTF-16 or UTF-32 (http://www.ietf.org/rfc/rfc4627.txt).

code:
curl "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json" \ | xxd - | less ... 0000090: 3a22 3230 3133 2d30 362d 3239 222c 2261 :"2013-06-29","a 00000a0: 7574 686f 7222 3a22 5a6f 6c74 e16e 2053 uthor":"Zolt.n S 00000b0: f366 616c 7669 2028 5a38 3629 222c 2265 .falvi (Z86)","e


The parser I was using was unhappy until I manually scanned for the offending bytes and converted to UTF-8 prior to unserializing the JSON into objects.

I can switch to using the XML version of the API, but I would prefer to use the JSON version if possible.

For what it's worth, I found a few decent pages [3][4] showing byte encodings for UTF-8, if anyone needs them for future reference;

[1] http://www.doomworld.com/idgames/ap...on=get&id=17259
[2] http://www.doomworld.com/idgames/ap...=17259&out=json
[3] http://www.utf8-chartable.de/
[4] http://dev.networkerror.org/utf8

[edit] added valid encoding types for JSON per the RFC and a link to the RFC[/edit]

Old Post 09-16-13 07:46 #
spicyjack is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05


Whoa. Lucky for you I decided to check up on this thread!

I'll take a look. Shouldn't be too terrible. The conversion is happening for the XML version, but I guess that I forgot about it for JSON.

If you need to get in touch, I'm reachable by the E-mail address in the ABOUT call.

Old Post 09-19-13 16:30 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
MTrop
Member


Posts: 421
Registered: 12-05


Okay. I've sent off the change to bloodshedder.

If you see the date at the top of the API page changed to 9/19/2013, it's been applied.

EDIT: Okay - looks like we're good here. Not sure if PHP is sending back the correct encoding headers, but according to Fiddler2, it's readable as UTF-8. Are you able to double-check?

Last edited by MTrop on 09-20-13 at 16:08

Old Post 09-20-13 03:38 #
MTrop is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
spicyjack
Mini-Member


Posts: 61
Registered: 08-09



MTrop said:
Okay. I've sent off the change to bloodshedder.

If you see the date at the top of the API page changed to 9/19/2013, it's been applied.

EDIT: Okay - looks like we're good here. Not sure if PHP is sending back the correct encoding headers, but according to Fiddler2, it's readable as UTF-8. Are you able to double-check?



Cool, thanks for the super-quick turnaround, I see the updated script live now.

Here's what I get now:

code:
wget -O - "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json" | xxd | less {snip} 00000a0: 7574 686f 7222 3a22 5a6f 6c74 c3a1 6e20 uthor":"Zolt..n 00000b0: 53c3 b366 616c 7669 2028 5a38 3629 222c S..falvi (Z86)",


So the 0xe1 is now 0xc3 0xa1, and 0xf3 is now 0xc3 0xb3, both of which are the valid UTF-8 replacement sequences for the offending ISO-8859-1 characters (according to http://www.utf8-chartable.de/).

This was actually a good lesson for me, it reminded me that I ALWAYS need to validate my input from external sources :) If I find anything else wonky, I'll be sure to let you know.

As far as the headers:

code:
$ wget --spider --server-response "http://www.doomworld.com/idgames/api/api.php?action=get&id=17259&out=json" Spider mode enabled. Check if remote file exists. --2013-09-21 11:17:25-- http://www.doomworld.com/idgames/ap...=17259&out=json Resolving www.doomworld.com... 38.68.5.148 Connecting to www.doomworld.com|38.68.5.148|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Sat, 21 Sep 2013 18:17:22 GMT Server: Apache/2.2.23 (Unix) PHP/5.4.10 X-Powered-By: PHP/5.4.10 Keep-Alive: timeout=15, max=500 Connection: Keep-Alive Content-Type: application/json Length: unspecified [application/json]


No, no encoding type mentioned in the headers. For what it's worth, the XML output also doesn't have encoding type in the headers, but since character encoding is part of the document, it's not such a big deal.

I'll keep in mind the contact e-mail address mentioned above in the about call, however, any chance of posting the API script somewhere publicly, preferably some place with a issues/bug queue, like GitHub/Bitbucket/Sourceforge/etc.?

Thanks again!

Last edited by spicyjack on 09-21-13 at 19:56

Old Post 09-21-13 19:29 #
spicyjack is offline Profile || Blog || PM || Email || Homepage || Search || Add Buddy IP || Edit/Delete || Quote
All times are GMT. The time now is 09:34. Post New Thread    Post A Reply
Pages (2): [1] 2 »  
Doomworld Forums : Powered by vBulletin version 2.2.5 Doomworld Forums > Misc. > Doomworld News > /idgames Archive Database REST API Now Available

Show Printable Version | Email this Page | Subscribe to this Thread

 

Forum Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is OFF
vB code is ON
Smilies are OFF
[IMG] code is OFF
 

< Contact Us - Doomworld >

Powered by: vBulletin Version 2.2.5
Copyright ©2000, 2001, Jelsoft Enterprises Limited.