I want to make a database of Doom YT videos - what kind of features would you expect?

I'm learning NodeJS, jQuery and similar technologies for web development, and I'm going quite well, I can even use the Youtube API, and started wondering on the creation of a crawler that collects videos with the word "Doom" in the title, through the related videos.

Originally, I made this to collect only UV- and NM-Speed or -Max videos, but depending on your demand, I may create an entire database for various Doom video searching purposes, well, because Youtube search is a pile of stinky garbage in the matter. The crawler I made collects basic data (link, title, uploader, etc.) but I can easily expand, and with a long enough list, additional data might be worth to gather and store in a proper MySQL database.

Some features I'm thinking of:
- search videos by: title, length, uploader, subject (Doom/Doom2/Plut/TNT/WADs), map, other?
- add additional data to videos (actual name of the map played, how many maps are shown, what kind of video it is (UV/NM/Speed/Max/Tyson/pure gameplay/commentary/etc))
- removing deleted/missing videos from the list
- constant expansion by automatic search in the chain of "Related videos", instead of regular search
- separating Doom, Doom 2, Final Doom, Heretic, Hexen, Strife, Doom 3 and mods specifically made for these

The crawler itself works, but yet there's no database to save the content into, so I'm gonna show it only later.

My question is: what do you think of the idea, and what should or shouldn't I add to such a website?

--------------------------------------------

Proposals so far:

- current world record for any map of any WAD (scifista42)
- labeling mapping tutorial videos as well (ChekaAgent)
- search for player, not only uploader (kuchitsu)
- quality label; original/PWAD music vids; collection search (GoatLord)
- categorizing Multiplayer videos too (from Reddit)

Feature suggestion: The user enters name of a wad, map number, and a speedrunning category, and the database should be able to return either EXACTLY ONE video showing the ACTUALLY CURRENT world record of that map in the given speedrunning category, or NO VIDEO and say that a video of the current world record doesn't exist.

I think it would be a good idea to make a huge collection of mapping/editing tutorials..

Scifista, this is a great idea, though very hard to implement since such a query can search only those videos where IWAD, PWAD and map number are all defined. Not even taking segment runs (Scythe 2) and D2ALLs into account. But it's a definitely very interesting challenge!

ChekaAgent, nice idea, a bit easier but it's quite hard to get all the tutorial videos based on an intuitive description. Gonna check out, how many of such videos are out there.

Remember that uploader is not always the same person as the player in the video. If you could extract the actual player's name from videos like this one, that would be nice.

I'm aware, but I'd keep duplicates in record - it's essential because in case of YT user LonerD, there are a lot of duplicates.

Extracting the player's name is ridiculously easy in the case of Skepticist, since needs only getting a substring out of the title (which is obviously string), or in case of ZeroMaster (who uploads only his own recordings, as far as I know), but is often impossible to predict from the metadata.

I may try my best to solve this problem, but it's very likely that some sort of manual edit is necessary here. Which needs a report system, but that also sounds quite essential for this task.

My videos should be quite easy to get in there, as I always fill in a template of data in each videos description, specifically: PWAD name, Map Number and Name, Time to complete, Category of the Run, Port Used including Version Number, and less importantly a link to the DW thread and my private host for my demos.

A lot of users do a similar thing, Nevanos and ZeroMaster do for sure iirc, and I think a couple of those channels that upload other speedrunners runs usually define whose run it is in the title or description. Of course you probably knew all this already, but I think if there's a possibility of making a standard for your crawler all this relevant info could be included in any future uploads from new youtubers so that they can be filed into the searches relatively easy.

Excellent idea. In addition to what was mentioned, searching by upload date, length and video quality would be good. Soundtrack videos as well. Get into some obscure stuff too, if say, someone wants to see IWADS from Maximum Doom or D-Zone.

Rayzik: I know, I'm watching your videos when I can spare some minutes for the aim, and I fucking bless your name, especially as a wannabe programmer, for doing that in such an organized manner.

Sadly, used port is rarely mentioned, I'm lucky if I can search the word "Boom" at all with a realistic chance to extract it from the description or the title. Same goes for category and DW link.

The problem is that video descriptions don't always contain useful information, leaving certain fields empty in some cases.

GoatLord: good ideas; date and length are possible, I'm unsure about quality but I can safely assume the API has such an option. Soundtrack videos is a great category idea; I even have some starting points. Maximum Doom & D-Zone vids are possible to find, since they are extremely likely to appear in the description or the title.

Progress has been made!



I'm beyond the basic issues, and going to attempt slowly implementing your ideas and desires. Categorizing will be made through the frontend UI, once I can write a general function, or set of functions for that.

Are you guys still interested?

This is/would be awesome; looking for videos to watch on Youtube can be pretty lame, sometimes...this seems like it would help alleviate that issue to some degree. Good luck with it.