/idgames Archive Database REST API Now Available

Doomworld Forums : Powered by vBulletin version 2.2.5

Doomworld Forums > Misc. > Doomworld News > /idgames Archive Database REST API Now Available

Author

All times are GMT. The time now is 20:57.

MTrop
Member

Posts: 275
Registered: 12-05

The /idgames Archive database now has a public-facing REST API which will enable programmers to write applications that can interface with it. The documentation for the API is available here.

12-14-12 16:57 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

GhostlyDeath
Forum Retard

Posts: 1012
Registered: 08-05

Neat, i'll have to code something up for this.

12-14-12 20:20 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

printz
CRAZY DUMB ZEALOT

Posts: 8193
Registered: 06-06

Does this mean there'll be mobile apps for /idgames? :)

__________________
Available for download: Automatic Wolfenstein - Wolfenstein 3D bot
MBF configuration for WHackEd2

12-14-12 21:15 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

kristus
Megablast!

Posts: 10289
Registered: 07-00

Wouldn't hurt, since the /idgames page works poorly in mobile browsers.

12-14-12 21:36 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

printz said:
Does this mean there'll be mobile apps for /idgames? :)

exl already made this app for Android Phones. It's pretty spiffy.

12-14-12 21:36 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

AlexMax
Senior Member

Posts: 1076
Registered: 01-03

This API looks fantastic. Nice work.

However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

Again, fantastic work...it's just probably not a good idea to call it "REST".

12-14-12 23:39 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

Use3D
Forum Legend

Posts: 4660
Registered: 04-02

Thanks for going out of your way to do this, what a great idea.

12-14-12 23:48 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

kb1
Junior Member

Posts: 234
Registered: 11-06

This is very awesome! A great idea - it's similar to something I've been working on at home, to organize my own archives :)
I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

Once again, very nice - good job!

12-15-12 01:11 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

AlexMax said:
This API looks fantastic. Nice work.

However, calling it a REST API is kind of a misnomer. REST API's usually don't have a single endpoint, preferring a structured URL hierarchy. REST also makes heavy use of HTTP verbs, and this API seems to not care about which HTTP verb is used, instead delegating operations to the action parameter. This looks to me more like some sort of RPC that uses query parameters instead of a JSON or XML payload.

Again, fantastic work...it's just probably not a good idea to call it "REST".

Yeah, you're right. It's not SOAP, and it isn't quite REST, but it's stateless, and some kind of in-the-middle thing. Maybe I'll just call it, what, a "Public API?" Get rid of the specific model type?

kb1 said:
I'd like to see another function: GetLatestID, which would return the highest id number. I suppose I could derive it from "getlatestfiles", but it's a bit more work, and, tricky if no new files are within the date range.

Once again, very nice - good job!

If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".

Last edited by MTrop on 12-15-12 at 01:17

12-15-12 01:12 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

kb1
Junior Member

Posts: 234
Registered: 11-06

Mista_T said:If you are looking for a date or length of time into the past, I think that would be doable. I'd probably put it in latestfiles, but as an alternative to "limit".

I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.

12-15-12 02:19 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

kb1 said:
I was kinda hoping to get the actual last id number (you know, something like SELECT Max(id) AS LatestId FROM Archive). I'm trying to do a periodic update of a local database I maintain. In this scenario, I would know that, the last time I ran my updater, the last id I captured was, say, 18000. So, today I might run a "GetLatestID" call, which might return 18105. I would then know I needed to download descriptions for 18001 thru 18105.

It could be done by date, if necessary, but, if given the choice, I'd prefer to use the id. Then, if I had to, I could run 2 updates on the same day, without having to check for duplicates in my local database, etc. More of a convenience than anything.

You DO get the ID number from the XML. latestfiles with a limit of 1 will get you that.

Would you also like a range? Like from an id number?

12-15-12 03:01 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

12-15-12 09:33 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

boris
meow

Posts: 3838
Registered: 05-00

DaniJ said:
Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

I concur.

12-15-12 10:30 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

exl
Personal Test Dummy

Posts: 521
Registered: 05-02

DaniJ said:
Any chance of getting JSON output in addition to XML? XML is great and all but I would much prefer JSON for this sort of thing.

Seconded. In writing the Android app I spent far too much time writing an XML parser that works at any decent speed. I don't know if using Android's built-in JSON parser is going to be much faster, but I wouldn't have to maintain 12 Kb of mostly if-elseif code for it.

12-15-12 10:59 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

fraggle
Filled with the code of Doom

Posts: 6607
Registered: 07-00

Make this a fourth vote for a JSON option.

12-15-12 16:13 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

Graf Zahl
Why don't I have a custom title by now?!

Posts: 7482
Registered: 01-03

JSON fifthed.

12-15-12 16:19 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

JSON output was something I was planning on doing. Now that there's demand, it will probably be the next thing I do. I hope that PHP (and Doomworld's PHP version) have something that will make this simpler. If not, though, it wouldn't be too hard to write.

Last edited by MTrop on 12-15-12 at 16:40

12-15-12 16:24 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

AlexMax
Senior Member

Posts: 1076
Registered: 01-03

Doomworld is running PHP 5.3.10. I would be very surprised if it didn't have JSON support compiled in.

12-16-12 00:13 #

Profile || Blog || PM || Email || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

AlexMax said:
Doomworld is running PHP 5.3.10. I would be very surprised if it didn't have JSON support compiled in.

Yeah, but it didn't have some recent DB connection libraries compiled into it so I'm not holding out hope...

EDIT: "Recent" meaning the last couple of years. YEARS.

12-16-12 00:31 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

Houston, we have JSON. I repeat, we have JSON.

Also, that little "latestfiles" extension too. Feed it a "startid" parameter in order to tell it what id to start at if you don't want it to start at the latest one.

12-16-12 07:44 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

printz
CRAZY DUMB ZEALOT

Posts: 8193
Registered: 06-06

Mista_T said:
Houston, we have JSON. I repeat, we have JSON.

Johnson and Johnson.

__________________
Available for download: Automatic Wolfenstein - Wolfenstein 3D bot
MBF configuration for WHackEd2

12-16-12 07:55 #

Profile || Blog || PM || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

DaniJ
Senior Member

Posts: 1958
Registered: 08-03

Mista_T said:
Houston, we have JSON. I repeat, we have JSON.

Excellent. I'll start prototyping some of the applications I have planned and let you know if I encounter any use cases which I feel aren't adequately served by the API. Though my initial thoughts are that it looks pretty good.

12-16-12 10:11 #

Profile || Blog || PM || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

I've just noticed that there may be some instances where a PHP array of one object may be returned as an Object and not an Array with one Object in it in JSON, so that may require a tiny bit of extra coding should you choose to parse JSON in JavaScript or something else. I really, really wished that PHP had separated Arrays and Associated Arrays into two distinct types...

I'm kinda afraid to mess with it because that would mean I could end up breaking structure for one output format versus the other, so I'm holding off unless it becomes a greater issue.

12-17-12 01:21 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

RjY
anARCHy

Posts: 860
Registered: 05-02

Some suggestions / bug reports

Insufficient backslash escaping in json output

Consider the description field returned when querying eaeuro03 (id=11518)

code:"description":"A fairly detailed map for Doom2. Inspired by
Fredik Johansson's incredible Vrack series. This map may cause a lot
of problems on slower computers. If your computer is slower than PII
200 Mhz, don't even try to play this level, sorry :\",

Notice how the smiley at the end escapes the closing quote? This breaks both json parsers that come with perl 5.14 and python 3.2.

not all fields are always defined

The documentation claims that several fields in returned search queries are always present when they are not. Something like comsw (id=12) or any file lacking an accompanying .txt (e.g. everything in themes/DAC/dac2005) have none of these fields.

index.php in url (cosmetic)

If it is easy please consider not appending "index.php" in returned urls.

code:
http://www.doomworld.com/idgames/index.php?id=

can be shortened to

code:
http://www.doomworld.com/idgames/?id=

which looks neater. index.php is just noise really.

12-20-12 21:10 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

MTrop
Member

Posts: 275
Registered: 12-05

RjY said:
Some suggestions / bug reports

Insufficient backslash escaping in json output

Fixed. Came about because preg_replace was being weird about escaping backslashes in regex for some weird reason. Was easier to do str_replace instead.

not all fields are always defined

Done-ish.

"Blank" fields will be "null" valued or whatever the DB is returning. I keep forgetting that undefined != null in JavaScript. Of course, as always, what is available is what the idGames parser picks up from the TXT when it is added to the archive. Disclaimer added to "search" function, just like "get" and "getfiles".

index.php in url (cosmetic)

Done.

Told Bloodshedder about new fixes. If you see "Updated: 12/21/2012" on the API page under the title, it's been applied.

EDIT: Found another string escaping bug that needed fixing. Should be fixed soon.

Last edited by MTrop on 12-21-12 at 19:08

12-21-12 07:32 #

Profile || Blog || PM || Email || Homepage || Search || Add Buddy

IP || Edit/Delete || Quote

All times are GMT. The time now is 20:57.

Doomworld Forums > Misc. > Doomworld News > /idgames Archive Database REST API Now Available

Show Printable Version |

Email this Page |

Subscribe to this Thread

Forum Rules:

You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

HTML code is OFF
vB code is ON
Smilies are OFF
[IMG] code is OFF

< Contact Us - Doomworld >