Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
sun_stealer

AI research and Doom multiplayer

Recommended Posts

Posted (edited)

Hi, Doom community!

 

I am a computer science Ph.D. student doing research in deep reinforcement learning. Some of you might know that Doom is a fairly popular platform amongst us thanks to https://github.com/mwydmuch/ViZDoom

In the last few years (`16-`18), even an AI competition has been held where bots competed against each other in FFA deathmatch. Despite the best efforts of multiple teams none of the participating agents were able to match a human player. This is what makes reinforcement learning research using Doom pretty exciting: if best AI agents we can make lose to humans even in this relatively simple game, there must be something we're missing (whether it's temporal/spatial awareness, memory, or something else)

Just as a clarification: here I am talking about agents that use raw screen pixels as input and that are trained from experience using machine learning, not the hand-programmed scripted bots we all know and love.

 

I am interested in implementing state-of-the-art RL algorithms to create robust Doom agents and I want to benchmark them against good human players. In particular, I am interested in deathmatch and duel scenarios.

 

I am completely new to Doom multiplayer and I am asking the community to help me create a fair benchmark.

First of all, I want to see with my own eyes what Doom multiplayer looks like (is it still alive?) What are the most representative maps and Doom versions that people use for duels and deathmatch?

I figured the most popular port is Zandronum, but there's a lot of stuff that I'm a bit lost. I need something that everyone can agree is "standard" Doom multiplayer experience, e.g. something like de_dust2 in CS or ctf_2fort in Team Fortress 2. Something that literally every Doom player is familiar with, something that is used in tournaments and casual games.

 

On top of that, I am asking whether people would be interested in participating in a small tournament against AIs, which will provide the information for a research paper?

Edited by sun_stealer

Share this post


Link to post
Posted (edited)

I would look into this if you're looking for the maps commonly played. Doom multiplayer is still alive, for sure. There are various discords you can look at such as MDF. In regards to ports - zandronum and Odamex are both used.

 

If you are looking for the dust2 or q3dm6 of Doom I'd have to say it's either doom2.wad map01 or dwango5.wad map01.

 

Edit: I should also say that this is very interesting and I wish you luck. I've thoroughly enjoyed watching the OpenAI stuff and would love to see something similar applied to Doom.

Edited by xvertigox

Share this post


Link to post
Posted (edited)

Hi,

 

There's a number of factors at play when picking what is "standard" (and there are many competing views on that). Easiest way would be to demarcate a line between oldschool (which has a more clearly designated "standards" or flagsets) and newschool (a bit more nebulous and has a variety of approaches). The most important distinguishing points between these 2 would be the enabling of jump, freelook, and finite actor height.

 

Oldschool mostly tries to emulate vanilla play in more modern ports such as Zandronum, though the other alternative choice would be Odamex. Odamex is arguably more suitable for oldschool play, and the maps that have been used in tournaments (most recently quake con I believe) would be SSL2, Dwango5 Map01, Judas23_, Doom2 Map01, and I think King1. Might've been Dwango5 map07, which is another viable standard map everybody knows. Those are for 1v1.

 

For FFA deathmatch, this question has actually been asked before on the zdoom forums, here: https://forum.zdoom.org/viewtopic.php?f=4&t=56552 which is mostly unchanged, the only difference being an additional newschool set made by the same people who made DBAB and Aeon, called NeonDM

Port choice is player preference, neither are "standard" but it's worth keeping in mind there are nuanced differences between them that changes feelings for players (Odamex is based on a much older version of ZDoom and CSDoom, while Zandronum is based on a much newer version of Zdoom). Zandronum supports more modern features which change map dynamics (such as 3D floors which allows room over room creation) while Odamex more definitely caters to old school

 

Newschool has less bound rules but there are many people willing to help you out, starting with the MDF yea. As for finding players, MDF and WDL are good places to look, I think Devastation is still around, Ralphis is still here, @dew can probably give good suggestions for viable players who aren't too immature for the research process.

Share this post


Link to post

@xvertigox @Decay thank you so much for your replies!

I see now, the difference between "oldschool" and "newschool" is very substantial indeed. The existence of freelook and jumping mechanic turns Doom into a completely different game! I'd be interested in using something that people play today, but this might be difficult if Vizdoom does not support it. I will check.

 

Meanwhile, does anyone have footage from recent Doom tournaments (duel, FFA)?

Best I could find, is this representative of Doom multiplayer in general? 

 

Share this post


Link to post

In my entirely personal opinion old school/classic Doom is the game to target. Once you add vertical mouselook, new weapons etc it changes the scope of the project enough that you'd be making an idtech1 agent which can encompass so many different weapons and map types.


The footage you showed is on point. If you'd like to see more recent footage I've linked a video below. You could also download Doomseeker and spectate matches yourself but it may be hard to find the specific kind of gameplay that you're after. There will be copious amounts of demos around that others could provide.

 

Spoiler

 

 

Share this post


Link to post

Your video looks much more exciting, maybe because of commentary. Thanks a lot! This is map01, am I right?

Share this post


Link to post
Posted (edited)

^ That's Dwango5 map01

 

I think this sounds really awesome and I for one would be happy to train our new robot overlords to kick our butts, or at least play a match or three :D

 

I think I also have to second what xvertigox said that you should prolly focus more on old school settings/maps to start things out, as the bots may not benefit from the added complexity of maps made in a new school style at the outset. I also want to note that Decay's reply is a great one.

 

Doom's mp is still alive but depending on time of day and day of the week you may or may not see matches going on.

 

If you plan on using Odamex then I'd recommend Doom Explorer; not sure if doomseeker works with odamex as thats a zandronum thingy but I could be mistaken on that.

 

There are a lot of little mechanics in doom that are useful wrt timing, planning shots and where to attack from. Certainly spatial awareness, where people can come from/go, spawn points, (spawn kills are a big thing in doom dm) places where certain weapons reign supreme, (like rockets in a long hallway) and other types of the more basic elements of strategy are good to have down, but there are also some lesser known things, or more-so really subtle things, such as bfg mechanics, or that hitscans like a shotgun usually hit better when fired from above. I'm not the best source of knowledge on this stuff by any stretch and I'm not sure how any of that would transfer over to a self-taught AI, but I think what you want to do is really cool and if you need a target dummy I'll be happy to run around aimlessly for a bit ^^

Share this post


Link to post

That video is Dwango5 map01, SSL2, and Judas23_

 

@Fonze Doomseeker supports odamex as well

 

Freelook and jumping are optional mechanics of course - the user does not necessarily need to use them in most maps, even in newschool ones. But I'd agree that oldschool is probably the best way to go for this particular project. Flags (settings) are different between ports so it'll be a matter of choosing which port you believe is best suited for the project (you could go chocolate doom for the most "pure" experience).

Share this post


Link to post

Yes, I agree with you guys, looks like old-school control style is the way to go. From what I've seen in the tournament videos, it definitely does not look like beating humans will be easy, actually, I expect humans to reign supreme for maybe another year or so.

DeepMind is doing similar work with Quake III, but even they couldn't dare to tackle the whole game yet: https://deepmind.com/blog/capture-the-flag-science/

This CTF mode is a very simplified version of Q3, without bunnyhop, without most weapons, etc.

 

Thanks for suggestions @Fonze, I will check Odamex and Doom Explorer now.

Share this post


Link to post

Very interesting topic! I thoroughly enjoyed watching the recordings of the prior AIs attempting FFA deathmatch. 

 

If you have any further questions on Doom multiplayer I would absolutely take heed of any advice @Decay can provide. He's a bonafide Doom multiplayer expert and I don't think there's anything worth knowing about it that he doesn't know. He's certainly my first port of call for any questions on modern Doom multiplayer. 

 

In terms of videos, I occasionally play "newschool" FFA with a few people on modern mapsets like NeonDM. They tend to be streamed, so you can watch recordings of such matches like here.

 

 

Share this post


Link to post

Hi sun_stealer,

 

Very interesting project.  I'd be interesting in assisting somehow, at the very minimum I can show you how to watch recordings (from the port.... not on Youtube!) and supply you with some great examples.  Funny enough, I am a chess player as well as a DM player... and your robots have already crushed us humans in that!

 

Once you've set an AI to learn for so many hours, I'd be interested in seeing some of the recording playbacks and give some feedback on what it has learned, and what it's still clueless on.

 

The interesting thing about reinforcement learning and doom, as opposed to chess, is how it considers scenarios.  Is it trying to adjust it's bias towards good things on a game by game basis... a frag by frag basis or a scenario by scenario basis?  Are you starting from ground zero of not knowing the controls, or are you starting it on the basis that it understands the controls?

 

Either way, I'd be interested in helping and also potentially playing in a tournament against AIs as you've stated.  If you're using classic rules and maps.

Share this post


Link to post
Posted (edited)

Hi, Devastation! Very good and relevant questions.

 

Quote

I can show you how to watch recordings (from the port.... not on Youtube!) and supply you with some great examples

 

That'd be awesome! If you can provide a few representative examples of high-level gameplay, I'd be very interested! Following the discussion above, I'll be most interested in "vanilla" Doom experience, at least without jumping and vertical aim. In particular, I'd like to see highly skilled deathmatch and duel gameplay on popular classic maps.

 

Quote

Once you've set an AI to learn for so many hours, I'd be interested in seeing some of the recording playbacks and give some feedback on what it has learned, and what it's still clueless on.

 

Sure, I will provide updates, and I think we can use this forum as a communication hub between me and the Doom community. This is working out great so far!

The potential training scheme will require billions of game frames of experience, which (at ~35 FPS) is on the order of years or even tens of years of playing Doom for the agent. In real time this will be days and weeks of training on a small GPU cluster. We don't have OpenAI resources to push it further than that, but it'll still be substantial.

Keep in mind that this is not a short-term thing, the expected timeline to be on the order of months.

 

I haven't really started working on it yet (finishing some previous stuff), but I already did some setup work and had my first "it's alive!" moment: 

Spoiler

 

This is the most primitive baseline algorithms training for just like 30 minutes agains easy bots, so far it only learned to shoot and generally stumble around. Has no clue about healthpacks etc, and doesn't have any memory, etc. But you have to start somewhere.

Quote

The interesting thing about reinforcement learning and doom, as opposed to chess, is how it considers scenarios.  Is it trying to adjust it's bias towards good things on a game by game basis... a frag by frag basis or a scenario by scenario basis? 

 

Training is usually is in so-called "episodic setting", in our case I guess one episode will be the entire duel/deatchmatch, and the final objective is to win the match. The agent will try to steer the future towards more favorable states (higher probability of winning, e.g. higher score, better position). The major difference compared to chess is that environment dynamics is considered unknown, we don't use Doom engine to consider different branching probabilities into the future, basically the agent has to figure out how the game works and how to plan directly from pixels by playing the game. Also Doom in partially observable, which means true state is fundamentally unknowable to the agent, therefore memory and awareness are very important. The chess AI, on the other hand, plays each move separately and independently (AKA Markov process).

 

Quote

Are you starting from ground zero of not knowing the controls, or are you starting it on the basis that it understands the controls?

 

I think from scratch is more interesting, although we might experiment with bootstrapping the agent from some human replays as DeepMind AlphaStar team did. Undecided at this point.

Share this post


Link to post
Posted (edited)

Sounds good.

 

I can provide you with all of the games from Quakecon 2013 (between myself, Demonsphere and JKist3) in demo format to view.  It's probably good to see a good player against an "average" player too, to gain a sense of how things can go wrong.  I'll dig up some historic games over the years.

 

You are stating that everything is done via pixel, are you allowed to use sound?  Sound is a major component of DM... if the bot is not allowed to base anything on sound, I doubt it could ever compete properly.  It will also need a way of recognizing and knowing the map, for backwards perfect movement, and timing.

 

A good example of timing in DM is the way one moves.  There is generally a tradeoff in DM movement, perfect moving (which is fastest) has the least visibility, so if you run into an opponent, you are in a bad spot.  Less optimal moving may be slightly slower, but grants the best visibility to be prepared if you run into an enemy.  If I know my opponents current location (thought sound or otherwise), and know that they cannot get to a specific location in a set amount of time, one would use perfect tight movement to get there as fast as possible.  If a player position is unknown, one would choose the safer movement style.

 

Some of these things may be hard to learn if an episode is an entire game.  There are many pieces in a game that can promote good play or bad play.  Think about yourself as a gamer.  If you get a frag, you generally say "wow I did something good", and if you die you generally think "I did something wrong".  Maybe an episode should be at least considered on a frag by frag level.  Do more of what gets me frags and do less of what gets me killed.  Going more advanced, in human terms, the outcome of many situations is actually a better or worse game state position from a given scenario, even if a frag was not achieved, such as dealing only 20% of damage but receiving 80% of damage.

 

Below is a list of things that every good DM player should be able to do, and some thoughts of how an AI could maybe learn these:

 

Good movement.  Computer likes when it moves and does not hit a wall, computer dislikes when it moves and touches a wall.  There also needs to be a way that the computer knows the level, so it can run backwards through the level without touching a wall.

 

Fast movement (sr40 and hopefully eventually sr50).  Computer detects pixels to gauge it's in game speed.  Computer likes when it moves faster, dislikes when it moves slower.  I assume eventually the computer would start randomly pressing enough buttons to figure out sr50.

 

Aim.  Computer likes when it fires and sees blood splats *OR* hears pain groan (sometimes you hear pain but cannot see it).  Computer dislikes when it fires and does not see blood splat or hears pain groan.  Computer should be weighted to liking more blood splats, it should eventually pinpoint the proper pixel to fire on for maximum damage, and hopefully understand the distance between players.

 

Damage.  As above, the computer can estimate damage based on blood splats and should like seeing more blood splats.  This will weight it towards the proper weapons in various scenarios as well.  It should know roughly how much damage was dealt in total over various situations, and how much health an enemy has left.

 

Damage balance.  This is difficult to describe, but it's knowing to fire at the closest range possible, just before your opponent reloads.  The basic premise in doom SSG battles if that you rush in while opponent is reloading, then your shot and then back away when you're reloading.  I have no idea how to help an AI learn this, but maybe the above notes on damage will just make it understand.

 

Sounds.  The computer needs to be able to pinpoint an exact location of an enemy and their *direction*, given it hears a sound (landing thud, damage, reloading shot, etc.).  This requires map knowledge.

 

Enemy timing.  Given a location of an enemy against passing time, the computer should know the furthest an enemy could have gone and the likelihood that it can happen.  This requires map knowledge and also how fast an enemy generally moves to get from any given location to another location.

 

Opponent state.  Computer needs to know weapons an opponent has for sure, weapons opponent likely has, the current ammo an enemy has for rockets and BFG, the enemy's current health.  The computer should be varying it's play eventually based on these game states.

 

Respawn whoring.  It needs to somehow figure out what optimal place to be when an opponent has just died, it should instantly start moving towards that direction given the known spawn points of the map.  It should be varying this based on current state (health, weapons).

 

Those are the basics.  If the computer could learn all of those things, it'd be about an average DM player, maybe above average given that it could probably aim perfectly.  Going more advanced, it'd need to learn how to manipulate it's own sounds to confuse or trick an opponent, how to guess an opponent's future plays and actions by associating it to previously seen actions, how to vary it's play away from the "best" play at times in order to stay unpredictable, and some other stuff.

 

Are you allowed to throw the AI into different maps for various periods of time to learn specific things?  For example, D5M18 would actually teach it much of the aiming and range that it requires to SSG battle properly.  You could also throw it into a map of just walls in order to get it to learn movement via not touching a wall and detecting it's speed.  It would hopefully learn how to strafe properly and know it's direction and where to go by being able to detect wall vs non-wall.

 

Also, some DM maps are simpler to learn.  For example, ssl2 is much more to do with aim, good movement and weapon control and not as much about determining an opponent's position using logic like D5M1 is.

 

Anyhow, hopefully this is of some help and gives you an idea of what you're getting into if you're trying to create a human-beating bot.  It also depends on your end goal, do you want it to be an average DM/tournament player?  Just be able to beat casual players?  Top notch gives the world elite a run for their money?

Share this post


Link to post

For some reason I always thought it would be trivial to make an "invincible" (or at least highly annoying) bot simply by exploiting its inherent strengths vs humans, like zero reaction time, internal knowledge of the game engine, dead-on accuracy with hitscan weapons etc. That's how AI opponents have been programmed for years, after all: with what to a human player would appear as borderline cheating (aka My Rules Are Not Your Rules). Come to think of it, a superior human opponent might appear to be just that [a cheater] to a lesser one ;-)

 

A good example where this was (ab)used is none other than Xaero from Quake 3 Arena. He is pretty much undefeatable unless you manage to find a hole in the AI and spawncamp him.

 

However, real-life DM games are often a super-brutal affair, and the top-ranking players in ZDaemon at least (e.g. Derrida) seem to have superhuman-like reflexes, never miss a SSG close-up shot to your face, take every opportunity to even lightly pelt a target from a distance etc. and in the end it's a contest about who'll rack up the most kills. The top players needn't even duke it among themselves for that to work (unless it's a 1-on-1). And let's not mention all those maps where the "gameplay" is reduced to camping/spamming the BFG and/or countering said camping/spamming. There's a lot of metagaming involved here.

Share this post


Link to post

Would love to help you research this (i.e. duel the hell out of Dev and others). Odamex doesn't have full servers all day, but it has a strong oldschool duel community that would probably love to help. It's easiest to round people up here: https://discord.gg/ZuwmxX2

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×