demo database ideas

[Schneelocke]

[Split off from this thread.]

Kristian Ronge said:

Indeed not. IMHO, Speed can be very important, depending somewhat on categories (espescially when it comes to barriers.. getting a time below x:00 for instance) but I hardly consider it the end of the world if a 2% faster demo of a long Tyson or something isn't in the archives. After all, those demos are COMPET-N records, not world records. ;-) But hey, maybe it's just me.

*noddles* Is there a site that provides world records etc.? I've kinda been thinking about a site to complement the SDA etc. myself; it'd be database-driven, like Compet-N is, but without any restrictions, so anyone could put up their demos there. The demos would not be hosted there; instead, the site would only contain entries with all vital (and non-vital) stats, including links to the actual demos.

I think it'd complement the SDA quite nicely, especially since I think the web interface to the SDA is quite awful (sorry, Opulent; I know you put a lot of work into that site, and I appreciate it a lot, too, but from a UI design point of view, it's a nightmare).

So far, it's just an idea I've been pondering, but it should actually be possible to implement it, too. I've got webspace which includes a MySQL database and all that, and I know Perl and PHP reasonably well at least, too, so it should be feasible.

The only question is whether it'd actually be accepted and used by the community. It wouldn't be nice to actually build a site like that only to find out that noone's using it, submitting entries and so on. ^_~

[Opulent]

Schneelocke said:

I think it'd complement the SDA quite nicely, especially since I think the web interface to the SDA is quite awful (sorry, Opulent; I know you put a lot of work into that site, and I appreciate it a lot, too, but from a UI design point of view, it's a nightmare).

So far, it's just an idea I've been pondering, but it should actually be possible to implement it, too. I've got webspace which includes a MySQL database and all that, and I know Perl and PHP reasonably well at least, too, so it should be feasible.

The only question is whether it'd actually be accepted and used by the community. It wouldn't be nice to actually build a site like that only to find out that noone's using it, submitting entries and so on. ^_~

this begs a response. :)
1) I completely agree with the poor layout and poor file access for the DSDA. It could be worse, but it could be a lot better.
2) I'm certain the big boys at Doomworld would be open to an SDA database. I would be willing to implement the changes with some technical assistance. Only problem here is it must be something I can update and modify on my own. Perl and mySQL are not completely foriegn to me, so maybe I could work with it.
3) Along those lines, would it be of value to add direct download links for every demolink (both competn.doom2.net and 3ddownloads links for each file)? Or is that a waste of time?
4) this 'other site' that was proposed sounds great to me. very Pdang-esque.
Perhaps all we need is that the SDA is managed better...
I'm open for suggestions and improvements. People weren't impressed with my CSS changes, so that died... I'm not sure what to do, so I do nothing. :(
help please.

[Grazza]

It's certainly a weakness that the DSDA site lacks direct links to the demos. The way I use it is that I find out the name of the file from the relevant page, and then find it in the file archive here. [Well, actually, since I downloaded the entire archive, I in fact look for it my local copy.] I don't generally use 3ddownloads because I don't have ActiveX, javascript or cookies enabled by default (sorry, too much malware about, and given that there is an alternative to changing by security settings each time, I'll use it). Casual users probably don't know about the FTP site though, and I do appreciate that it isn't guaranteed to be there forever.

It's easy enough to find a file once you know its name, since the archive is organized alphanumerically (apart from the TAS demos).

The ideal solution would be something like the /idgames database, but for demo files (i.e. the contents of ftp://competn.doom2.net:8002/pub/ would take in most of them). However, the /idgames database works by parsing the fullsort document and the textfiles accompanying each zip files. Presumably something equivalent to the fullsort could be constructed externally for the demo files, but all that could be indexed is the filenames, sizes and dates. Might be a lot of work for a small improvement over the current situation (i.e. once you know the filename, you don't need to browse the direcories yourself).

Anyway, anything that means that people new to DSDA are less likely to come to a grinding halt when they want to download a demo would obviously be a very positive development.

Maybe this discussion should be in its own thread?

[Schneelocke]

Opulent said:

2) I'm certain the big boys at Doomworld would be open to an SDA database. I would be willing to implement the changes with some technical assistance. Only problem here is it must be something I can update and modify on my own. Perl and mySQL are not completely foriegn to me, so maybe I could work with it.

Having the thing hosted on Doomworld would be quite cool, of course, especially since it'd mean the site wouldn't have to rely on my own webhosting (which really shouldn't be a problem, but you never know); and of course, it'd fit Doomworld quite well, too, and probably expose the site to a much wider audience.

Updating and modifying shouldn't be a problem. I would hope that there'd be a web interface for all interactions with the database; a query tool etc. for all (not logged in) users, a tool to submit new entries for registered users, and a full admin tool for, well, admins. ^^

It probably should also be possible for guest (i.e., unregistered) users to request an entry for a new demo (which would then be created by an admin; the entry, that is, not the demo) so demo recorders wouldn't be scared away by registration requirements.

3) Along those lines, would it be of value to add direct download links for every demolink (both competn.doom2.net and 3ddownloads links for each file)? Or is that a waste of time?

No, not at all. It would be great if you could just click on a demo entry to download it, just like you can at the Compet-N. Much, much more user-friendly.

4) this 'other site' that was proposed sounds great to me. very Pdang-esque.
Perhaps all we need is that the SDA is managed better...
I'm open for suggestions and improvements. People weren't impressed with my CSS changes, so that died... I'm not sure what to do, so I do nothing. :(
help please.

I really think that a database-driven site is the only way to go; everything else just gets unmanageable once you get past a certain size.

I'll be glad to help out in every way I can, too; for now, though, I think the focus should be on brainstorming and planning. I think the most important questions right now are:

1) Where would this be hosted?
2) What information should be included in the database, and what should the database layout look like?

I hope the answer to 1) would be "Doomworld"; as for 2), my idea was/is for the site to be as inclusive as possible, so it should be possible to record all necessary information, from time to kill/item/secrets % to skill level, flags, weapons used, special achievements (like "takes no damage", "pacifist") and so on.

The database layout should also be coop-demo friendly (and maybe dm-friendly? should dm demos be included?), as well as flexible enough to not impose arbitrary limitations that would come back to haunt us later. I have been thinking about a database scheme for a while myself (before mentioning this idea anywhere), but I'll have to go over it again before it'd be ready as a first public request-for-comment.

--------

Grazza said:

It's easy enough to find a file once you know its name, since the archive is organized alphanumerically (apart from the TAS demos).

Yes, but sometimes, finding the file that contains a specific demo isn't all too easy, either, especially when you have files with descriptions such as "20 maxdemos from XYZ" and so on, and when the files are (seemingly) unordered on the WAD's page.

The ideal solution would be something like the /idgames database, but for demo files (i.e. the contents of ftp://competn.doom2.net:8002/pub/ would take in most of them). However, the /idgames database works by parsing the fullsort document and the textfiles accompanying each zip files. Presumably something equivalent to the fullsort could be constructed externally for the demo files, but all that could be indexed is the filenames, sizes and dates. Might be a lot of work for a small improvement over the current situation (i.e. once you know the filename, you don't need to browse the direcories yourself).

That would already be an improvement, but what I have in mind myself is a site which would contain all the vital stats on demos; these would probably have to be entered by hand, but I don't think it'd be much more work than updating the SDA pages manually (as I suspect Opulent does these days), especially when you have a slick web interface to the database that allows you to enter information on new demos in a quick and intuitive way.

FWIW, I'm also wondering whether it wouldn't be a good idea to allow players to log in to the site and enter the info for their demos themselves if they want to. That way, the site would be updated continuously instead of seeing big but sporadic updates (like the SDA and Compet-N currently do), and there'd also be a lot less work for the admins.

[Grazza]

If the information needs to be entered by hand, then I really can't see this happening. Moreover, some of what you're proposing would need to be generated by rewatching all the demos in the archive and taking notes.

I think it is generally quite clear what the file names are. Normally each page is simple enough for it to be obvious; for more complex cases (e.g. HR) there are normally symbols by the demos indicating which pack they are in. I believe the most worthwhile change is to add direct links to the files. Anything beyond that would need the data to be more regimented to begin with (standardized filenames and textfiles - and clearly they are not).

There are over 1000 html files, weighing in at a total of more than 6 megs. A manual solution isn't realistic - it would need to be (largely) automated. Currently the links are of the type: <<b></b>a href="http://www.3ddownloads.com/?directory=/doomworld/sda/" target="new" pevl1954.zip">30tv3108<<b></b>/a>. These would need to be changed en masse to: <<b></b>a href=http://competn.doom2.net/pub/sda/0-a/30tv3108.zip>30tv3108<<b></b>/a> or <<b></b>a href=http://www.3ddownloads.com/doomworld/sda/0-a/30tv3108.zip>30tv3108<<b></b>/a> (etc.) or a combination of the two somehow. The tricky thing from the point of view of scripting this, is that the location isn't determined in all cases purely by the filename.

[Opulent]

hmm... some of this is certainly do-able... even without a new site (or a re-vamped DSDA).

one problem I have is that the DSDA is for about 30 people worldwide. it takes a lot of effort to do such grand changes for only 30 people.
for example: it is hard to get motivated for the DDD when you look at this:
http://www.3ddownloads.com/doomworld/sda/ddd/

I don't know what is needed as far as admin priviledges on the web server, so we need to work that out, then present this to Julian, Lut, and the admins...
One of the steps to running a website is to just do it.
Schnee, email me: opulent@cox.net

[Chain Mail]

Schneelocke said:

*noddles* Is there a site that provides world records etc.? I've kinda been thinking about a site to complement the SDA etc. myself; it'd be database-driven, like Compet-N is, but without any restrictions, so anyone could put up their demos there. The demos would not be hosted there; instead, the site would only contain entries with all vital (and non-vital) stats, including links to the actual demos.

That's nonesense, because most DOOM engines are different in one way or another, so comparing their demos is inaccurate and silly... at least, you must make lists for each engine, or demo version.

[Schneelocke]

Chain Mail said:

That's nonesense, because most DOOM engines are different in one way or another, so comparing their demos is inaccurate and silly... at least, you must make lists for each engine, or demo version.

I'm not sure what exactly your point is.

[Schneelocke]

Grazza said:

If the information needs to be entered by hand, then I really can't see this happening. Moreover, some of what you're proposing would need to be generated by rewatching all the demos in the archive and taking notes.

I know. But this would only be a problem for the initial phase of building the database; as soon as that's done, adding new stuff would be no more work than it is now.

[Schneelocke]

Opulent said:

one problem I have is that the DSDA is for about 30 people worldwide. it takes a lot of effort to do such grand changes for only 30 people.
for example: it is hard to get motivated for the DDD when you look at this:
http://www.3ddownloads.com/doomworld/sda/ddd/

I don't know what is needed as far as admin priviledges on the web server, so we need to work that out, then present this to Julian, Lut, and the admins...
One of the steps to running a website is to just do it.
Schnee, email me: opulent@cox.net

OK, I will. As for admin privileges, I doubt we'd really need any, except for access to a database (the term "admin" I used was only supposed to refer to people with full access to the new site, much like you're the admin for the SDA now, not to any kind of admin access to DW as such); and since every database server supports having several separate databases at once, this shouldn't be a problem, since we'd just create a new database that wouldn't interfere with any existing DBs used by DW.

The question of whether it's worth the effort is a good one, of course. I'm not sure how many people there really are that are interested in DOOM demos; if it really is only 30 or so, then it might not be worth the effort after all, but that number seems rather low to me.

Oh, and regarding the amount of initial work required to feed all necessary info about existing demos to the database... I'm wondering whether it wouldn't be possible to create a custom prboom version that dumps all necessary information after demo playback (without actually showing the demo, instead going through it as fast as possible). That way, it would be relatively easy to collect stats for a large number of prboom- or doom2.exe-compatible demos at least. I'm not a C wizard (in fact, I hate C), but I'm sure it would be possible.