Cool concept. Yeah, it might be impossible to do accurately, but even trying to think it through and seeing what other people rate as relevant is interesting.
One thing worth mentioning IMHO, Doom gameplay is varied enough you can make up for a deficiency in any area (or even a lack of twitch skill in general) through other means. I'd suck at every test proposed so far, yet I can tackle many of the hardest maps. Well, unless you stick me in a Plutonia-style megawad, in which case I just crawl in a corner and weep.
In the context of player reception to mapset difficulties, perceptions may be just as interesting as objective performance. Sometimes you find out the guy who thinks the map is too hard considers even a single death as failure, and the guy who thinks the map is too easy dies a dozen times but shrugs it off. Expectations probably vary per audience, I'd wager slaughtermap fans as a whole have a greater tolerance for mashing the space bar, and speedrunners probably even more than that; and per type of map as well, i.e. in a purely mechanical map people might be more willing to approach it as a puzzle to be solved and don't mind needing a few tries, whereas in a level telling a story, or experienced for its looks or atmosphere, getting killed at all, replaying areas of no value on a subsequent playthrough might be seen as a negative.
Ideally you'd want to just record and keep track of people while they play normally. Honor system works well enough for demos after all; but then the notion of "skill" might be too loaded and bring people to cheat, who knows. My cat jumped on the keyboard, I got a hand cramp, I was distracted because I thought of something else, just like in our appreciation of difficulty we differ when it comes to our definition of fairness as well. But on a personal level, I'd love to see a ZDoom/PrBoom+ fork tracking my stats locally and spitting out at the end of a level and/or a session, say, how well I did in terms of deaths, damage done, damage received, ammo efficiency, compared to my average stats in general and on that particular map.
Last edited by Phml on 12-23-13 at 21:29