larsboy Posted April 24, 2020 (edited) Doom Map Database is an application for analysing and storing information about content for Doom-engine games. Having been playing Doom on and off since 1994, I wanted a way to keep track of what I have played. Combined with wanting to learn more about the Doom engine and to do something in C++, this resulted in a hobby project I tinkered with over the years. Having now reached the state of features and maturity I am happy with for my own use, I have made a version 1.0 public on GitHub. I hope it can be useful to others as well. Key features: - Reads wad and pk3 files, and zips and collections of such files, producing a report on the contents. - Analyses level data, in original, Hexen/Zdoom and UDMF formats. - Produces gameplay statistics based on editable definitions of the map things. - Supports Dehacked patches and Decorate, to produce correct statistics for mods. - Key properties and stats are saved in a database, with entries for each map and each wad. - Stores a line drawing of each map. - Calculates square area of the maps, using the simple polygons of the BSP tree. - You can add screenshots, descriptions, ratings and other personal content. - Custom tags to label your map entries. - A People table to store map authors. - Text search, data views/filters and custom wad and map lists. - Generates extensive statistics from the database. Project on GitHub A Windows binary is available in releases. The application is based on wxWidgets, which is multi-platform. I only work on Windows, but I have tried to be platform-independent, so it shouldn't be too much work to get it to compile for Linux and Mac as well. The GitHub site has a user guide in the wiki section. The application now does what I want for personal use, and I don't currently have further plans. If others start using it I'm sure there will be feature requests and bug reports. I can set up the GitHub page to handle this if there is interest. If someone has experience with C++/WxWidgets (I hardly do myself!) and would like to contribute, that would be most welcome. UPDATE: DMDB v1.03 is released on GitHub, and includes a database of all wads of idgames/levels. Edited May 12, 2020 by larsboy 17 Share this post Link to post
larsboy Posted May 2, 2020 I have been applying my tool to a complete dump of idgames (from March 1st 2020). I'd like to analyse every file in levels. It is a good test, and it would be cool to get the statistics. Has anyone generated extensive gameplay statistics for all of idgames before? Like total monsters, total hp, total linedefs, total secrets, etc.? So far I have been able to process all of levels/doom, so all wads for doom1. I had to improve handling of errors and corner cases, so I've made a new release. Here are some key figures: 2656 wads* analysed, with 7029 maps (*wad here is technically the zip with everything in it) Geometry statistics Total Average Min Max Linedefs 8698327 1237.5 4 41954 Sectors 1602870 228.0 1 11757 Things 2127468 302.7 0 8381 Secrets 22211 3.2 0 51 Area (MU) 12.0 0.012 1057.069 Maps with computed area: 6933 Gameplay statistics (UV) Total Average Min Max Enemies 808177 123.9 1 8188 Enemy HP 11869091 1820.1 2 835056 Ammo ratio 5.58 0.00 5870.63 Health ratio 0.60 0.00 148.00 Armor ratio 0.77 0.00 953.85 Maps with gameplay stats: 6521 Also attached is a screenshot showing the breakdown by year. Most of the maps are from '94 (and obviously a few of the file dates are wrong!) If I can complete the analysis of all level wads, I will upload the database somewhere. 9 Share this post Link to post
holaareola Posted May 3, 2020 This is an extremely cool resource. I think this will be very useful. Not just for informational uses like finding trends in mapping, but also for intelligent filters in launchers. Interesting to see 2020 is so far setting to be a record breaking year across many metrics. We're in an era of excess, apparently. Awesome work, looking forward to the Doom 2 stats. 3 Share this post Link to post
WadArchive Posted May 3, 2020 I have some statistics on nearly 80,000 wads (not all have levels). I don't have area or HP but do have things, sidedefs etc.. 3 Share this post Link to post
elend Posted May 3, 2020 Wow, quite interesting actually. Can you generate graphs like "linedef increase as years passed on", "average amount of enemies per WAD in a given year" and stuff like that? That might be an interesting addition to "The Evolution of Doom Mapping" by NotJabba, as well.. :p 2 Share this post Link to post
larsboy Posted May 3, 2020 14 hours ago, holaareola said: This is an extremely cool resource. I think this will be very useful. Not just for informational uses like finding trends in mapping, but also for intelligent filters in launchers. Interesting to see 2020 is so far setting to be a record breaking year across many metrics. We're in an era of excess, apparently. Awesome work, looking forward to the Doom 2 stats. Yes, I thought it was interesting that 2019 had the highest number of Doom1 maps since 1996. Note that some of these might be total conversions, so not really Doom1. I also like to think of how long it would take to kill all those 808 thousand enemies, or find all 22211 secrets! And the Doom2 folder is almost 20 times as big, in MBs. 12 hours ago, elend said: Wow, quite interesting actually. Can you generate graphs like "linedef increase as years passed on", "average amount of enemies per WAD in a given year" and stuff like that? That might be an interesting addition to "The Evolution of Doom Mapping" by NotJabba, as well.. :p I can get a lot of statistics from the data. I haven't made views for everything, but more than is shown here. I don't have plans for creating my own graphs in the application, as there are other applications which can do that better, but I could add some export features. The database I am building here is only what is extracted with the automatic analysis. I will do some cleanup later, such as for the obviously wrong dates. But the application is also very much for entering data manually, which is what I do in my own database for wads I have played (screenshot, rating, comments, authors, style tags, etc). 3 Share this post Link to post
larsboy Posted May 4, 2020 On 5/3/2020 at 4:36 AM, WadArchive said: I have some statistics on nearly 80,000 wads (not all have levels). I don't have area or HP but do have things, sidedefs etc.. I checked out wad-archive.com, and it is very impressive.That is many times the content of idgames! How to you generate the screenshots? 0 Share this post Link to post
WadArchive Posted May 5, 2020 I created a script that runs in a virtual machine to fetch the wads and takes a screenshot. It doesn't work for all wads as some wads will depend on other wads or the engine/iwad guessing isn't 100% 2 Share this post Link to post
taufan99 Posted May 6, 2020 This looks great! I can see the appeal for the more dedicated fans, and this sure is one great achievement. 0 Share this post Link to post
larsboy Posted May 7, 2020 Some joker made a wad with 100,000 maps, as well as a 30k one. I'll leave those out of the database, or else the majority of maps are little squares. I've entered a lot of the Doom2 wads. The database so far has about 10,000 wads and 29,000 maps. I only do a little each night, and sometimes I have to improve the code to handle special cases, but it should be done in a few more days. 1 Share this post Link to post
WadArchive Posted May 7, 2020 Yea I had to put in some exceptions for those WADs so that it didn't take screenshots and the HTML is partially pre-built to speed up the page load 0 Share this post Link to post
DavidN Posted May 8, 2020 This sounds amazing - I’ve wanted to have the ability to preview how much grief a WAD will give me :) 0 Share this post Link to post
larsboy Posted May 9, 2020 I have finished entering all wads from idgames/levels into a database. The database has a total of 12,768 wads and 42,355 maps (this includes Heretic, Hexen, Strife and Hacx). There has been a few zips DMDB couldn't process even with some manual help - where the content wasn't packaged as a wad or pk3, or where map data was in an exceptional form it couldn't handle. There's even two maps for Doom64 EX, with map data in a format I haven't seen before (Doom64, I presume). But I think this is a representative data set. The screenshot shows some map statistics by year (a few wads have file dates outside this range). We see that 2019 was a very good year, with 1927 maps, the highest number after 1996! The low point was 2002, with 771 maps, after a gradual decline from 1996. The amount of content per map has of course increased a lot over the years as maps have gotten more detailed as well as bigger in terms of square area. From an average number of linedefs of 780 in 1994, its 3333 in 2019. Sectors likewise from 159 to 601, and THINGs from 291 to 683. Average square area (the unit here is 1024x1024 map units) has gone up from 10.9 to 28.6. Note also that the averages from 1995 and 96 are down from 1994, probably because a lot of the maps were for deathmatch. I will add some more statistics and filters to DMDB, and make a new release of the application. Then I will also make the idgames database available for download. 1 Share this post Link to post
taufan99 Posted May 9, 2020 Good job! Although I have to wonder if that much data affects the program's size. Must be quite huge, I guess. 0 Share this post Link to post
larsboy Posted May 9, 2020 1 minute ago, InDOOMnesia said: Good job! Although I have to wonder if that much data affects the program's size. Must be quite huge, I guess. Thanks! The application (DMDB.exe) and the database I have created with it are two separate things. But the core database - the wad and map tables - is actually very small, less than 5 MB in total. I have hand-coded the data model and storage to use as few bytes per entry as possible, and am not using any SQL engine. But it has generated a map drawing for each map, and these pngs are a few hundred MBs. It was a clear goal when making DMDB to be able to always have the whole core database in memory, so that all scrolling, filtering and statistics generation is instantaneous. I like that I can scroll through the list of 42k maps freely, with no lag. 3 Share this post Link to post
larsboy Posted May 12, 2020 A new release of DMDB is now ready on github, and the idgames database is available. DMDB version 1.03 now includes a version of the idgames database. Folder idgamesDB, just extract everything and select this folder when starting the application. Note that this version does not include the map images, to keep the file size reasonable for the application itself. A version with map images for all 42355 map entries is available from my Google Drive, here: https://drive.google.com/file/d/1Umo2YD62WRIGxOOFnvWg_3MIafeUe2V5/view?usp=sharing This database allows you to browse all wads and maps in idgames/levels, with the gameplay stats, statistics etc. Keep in mind that this is the raw output of the wad analysis, without human input, so not all data is perfect. You can also use it as a starting point if you want to keep track of what you have played, with own comments, rating, etc. The latest version of DMDB generates wad statistics, as in the attached screenshot. Here you can see the average maps per wad, which have climbed from 1.59 in 1994 to 8 in 2019. So these days there are fewer wads released, but with more maps in each. The average wad size has gone from 0.18 MB in 1994 to 11 MB in 2019. 3 Share this post Link to post