Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
Memfis

How to find all wads by year on idgames?

Recommended Posts

I tried searching for "zip" in the textfile and sorting results by date: https://www.doomworld.com/idgames//index.php?search&page=485&field=textfile&word=zip&sort=time&order=asc

 

But that doesn't seem to include everything. For example I don't see this wad in the results linked above: https://www.doomworld.com/idgames/levels/doom2/m-o/nevermo2 It's from 06/16/04 so it should be between 06/10/04 and 06/17/04 but it isn't?

Share this post


Link to post

Oh I know, it's because "zip" isn't actually in the textfile... What should I do then?

Share this post


Link to post

I'm assuming the api still works? https://www.doomworld.com/idgames/api/

 

shouldn't be too hard to write a script that crawls through the metadata of every upload, then sorts by date added to archive. there'd probably still be some weird cases or slight inaccuracies, but yeah...

Share this post


Link to post

Hi. so I ran this. I attached a sorted list of files, everything in the /levels/ directory, excluding the deathmatch/ sections.

 

Spoiler

import os, time, urllib2, json
import cPickle as pickle

BASE    = 'https://www.doomworld.com/idgames/api/api.php?action=getfiles&name='
D1      = ['levels/doom/','levels/doom2/']
D2      = ['0-9/','a-c/','d-f/','g-i/','j-l/','m-o/','megawads/','p-r/','Ports/','s-u/','v-z/']
PORTS_D = ['0-9/','a-c/','d-f/','g-i/','j-l/','m-o/','megawads/','p-r/','s-u/','v-z/']

if not os.path.isfile('allDat.p'):
	allDat = []
	for d1 in D1:
		for d2 in D2:
			if d2 == 'Ports/':
				for d3 in PORTS_D:
					myDir = d1+d2+d3
					myCmd = BASE+myDir+'&out=json'
					resp  = urllib2.urlopen(myCmd)
					allDat.append(json.loads(resp.read()))
			else:
				myDir = d1+d2
				myCmd = BASE+myDir+'&out=json'
				resp  = urllib2.urlopen(myCmd)
				allDat.append(json.loads(resp.read()))
	pickle.dump(allDat,open('allDat.p','wb'))

#
allDat = pickle.load(open('allDat.p','rb'))
sorted_dat = []
for i in xrange(len(allDat)):
	if 'content' in allDat[i]:
		for file_dat in allDat[i]['content']['file']:
			sorted_dat.append([time.strptime(file_dat['date'],'%Y-%m-%d'),file_dat['date'],file_dat['url']])
	else:
		print allDat[i]
		exit(1)
sorted_dat = [[n[1],n[2]] for n in sorted(sorted_dat)]

for n in sorted_dat:
	print '\t'.join(n)

 

 

sorted_dat.txt.zip

Share this post


Link to post

I'm not familiar with this kind of thing but what is the deal with the pickles? Importing pickles, pickle loads and even pickle dumps!

Share this post


Link to post

@Nevander pickles are just serialized python objects. it's a way to immediately save and reload arbitrary variables from disk without having to parse anything or having to use a specific format. They're only in that little script because my initial thought was that grabbing all the metadata from /idgames might take awhile, and I wanted to save it to disk so that I wouldn't have to do it more than once. it turns out that was completely unnecessary because it takes all of 10 seconds to harvest it all, but I didn't feel like cleaning up my code snippet before posting it so you get to see the sloppy version :p

Share this post


Link to post

Ribbiks, you can make the same sorted list for Heretic/Hexen wads?
 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×