Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

sun_stealer

Members
  • Content count

    16
  • Joined

  • Last visited

About sun_stealer

  • Rank
    Warming Up

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. sun_stealer

    AI research and Doom multiplayer

    What I might need soon is a way to progressively increase the difficulty of bots to keep them around the same level as the agent during training, until the agent is able to consistently beat the hardest ones. If you start training the agent from scratch on hardest bots it does not learn very well, because bots just kill it immediately.. Besides reactiontime, is there a way to modulate the difficulty of tdbots in such a fashion? Also, on maximum settings, are your bots stronger than standard?
  2. sun_stealer

    AI research and Doom multiplayer

    Oh, I am only using them cause they are already included into Vizdoom, it's a matter of effort. I am not questioning the quality of your bots :) Currently I have only one pair of hands working on this project (mine), so I just cut all corners that I can.
  3. sun_stealer

    AI research and Doom multiplayer

    Working on it right now :D
  4. sun_stealer

    AI research and Doom multiplayer

    That's awesome! I didn't know we have custom bots too. I will check it out! https://zdoom.org/wiki/Bots this also has information on how to set up standard bots.
  5. sun_stealer

    AI research and Doom multiplayer

    Some more intermediate results: Here the agent is training vs built-in scripted bots and has no problem outscoring all of them in 100% of matches. It definitely still has problems with the aim (this agent does not have access to the mouse, only turning buttons), and the overall strategy is very hacky. Hopefully, with self-play, we will be able to see more diverse tactics :) These bots are pretty lame tbh. I add them via addbot console command, and as far as I understand, this creates a bot with random characteristics (aim, movement, etc.) Does anyone have experience with Doom scripted bots? I would like to have it consistent (e.g. bots are the same across experiments for reproducibility) and potentially also make the bots progressively harder as the training progresses. These videos are just sequences of raw neural network inputs, therefore the resolution is so low. I gotta work on hires rendering of the gameplay, maybe via replays.
  6. sun_stealer

    AI research and Doom multiplayer

    Some footage from intermediate tests. Here the task is to just find and kill as many monsters as possible in limited time. I was testing the compositional action space here, basically this agent can press all the available buttons at the same time, while previous bots were only allowed one action at a time (for implementation reasons). Interestingly enough, the agent almost never runs out of ammo if pistol clips are regularly collected, therefore the agent didn't learn the concept of ammo conservation (at least not yet at this stage of training). This is reinforcement learning for you: you never know what kind of hacky strategy is going to emerge.
  7. sun_stealer

    AI research and Doom multiplayer

    Again, thanks for the input guys, I really appreciate it! Using about 100 hacks I was able to get this first self-play setup, where I will do my first multi-agent experiments: https://www.youtube.com/watch?v=dHGSZRFTnf0 These are 8 random agents (untrained) acting together in a single deathmatch, a scaffolding for the neural network training environment. I was able to collect about 11500FPS of Doom gameplay on a single computer, which means that the agents will be learning at about ~300x speed compared to real time. But this is only the beginning. @JKist3 I agree that duel should be the ultimate goal, and I anticipate this will be very hard for the current state-of-the-art agents. Which makes it that much more exciting, this is a very clear benchmark to beat! We're planning to focus first on like 8-player deathmatch which is much more reactive and chaotic and should be easier for the agents. Then we might start thinking about duels.
  8. sun_stealer

    AI research and Doom multiplayer

    Thanks a lot for opinions on this, I really appreciate! I agree that porting VizDoom functionality into other mods will not be a good investment of resources now. @andrewj can you please elaborate on: Is it unreliable/laggy/unplayable or is it just inconvenient to use because everyone should be present before the game starts and there's no server browser? I'd still love to be able to host my own server, even given the limitations, just because it is so much easier to set up AI bots locally. If I need to make a full local deployment (e.g. game with AI bots), I'd have to package the AI agents neural networks in some installable package, and that is a lot of effort, although much less than porting VizDoom. FYI, there's also a github thread on this https://github.com/mwydmuch/ViZDoom/issues/392
  9. sun_stealer

    AI research and Doom multiplayer

    I have a question for people who know how multiplayer works in different Doom source ports. As you might know, VizDoom (a platform for agent training that we're using) is based on a relatively recent version of ZDoom. On the other hand, Odamex is based on CSDoom, which is based on a very old version of ZDoom and features a different networking code (?) Do you have any idea whether it'd be possible to establish a multiplayer game between these versions? Is it possible to create a VizDoom (ZDoom) server that can be joined using Odamex client or something else modern? Will my server show up in the Doomseeker/explorer? EDIT: Ok, I read some more documentation, and it looks like Odamex and other CSDoom derivatives use a different, more modern approach to networking, with a proper client-server architecture. ZDoom uses ancient networking code from the original Doom open-source release. So I don't think those two really play well. Another annoying fact is that ZDoom requires all clients to be connected for the deathmatch game to start, which is not the case for Odamex. I guess I have only two options here: 1) port parts of the functionality of VizDoom into something like Odamex (e.g. python APIs, screen buffers, etc.) This is a big effort. 2) just leave it in ZDoom world and ask people to use something like GZDoom client to play against AI. This sucks because I wanted my server with AI agents to be always available in Doomseeker, which won't be possible. I can do #2 first and if there's interest and resources I can think about #1.
  10. sun_stealer

    AI research and Doom multiplayer

    Valid point. Let's be absolutely clear here, if we wanted to just build a human-beating Doom bot, we would do it via accessing the internal game state, hand-programming perfect aim, etc. This is definitely a nice technical exercise, but this is not AI research. What we want to do is to build an agent that can learn how to play the game from experience and has the same access to game state as humans do, e.g. screen pixels & sound. Achieving something like this is a tiny step on the way to general intelligence - an agent that can make effective decisions in the real world. Arguably, this "trained" bot should be also more interesting to play against. It is not hand-programmed so it should exhibit less robotic behavior, although should still be exploitable as it lacks general reasoning capabilities of a human and cannot quickly adapt. A good comparison can be AlphaStar StarCraft II bot versus in-game "Cheater" AI. A neural network based bot is definitely a more interesting opponent to play against.
  11. sun_stealer

    AI research and Doom multiplayer

    Hi all! I have fully switched to this project and I think we'll be working on it for at least 2-2.5 months from now. I trained a couple of very simple baseline agents just to get a feel of what I'm dealing with, and now I'm working on infrastructure for distributed reinforcement learning to be able to leverage many machines and many GPUs to train big models on more data. After this, the plan is to set up the self-play training where the agent battles against current and past copies of itself. This will be done with either embedded VizDoom multiplayer, or by modifying the engine to allow multiple agents to exist within the same process (not sure if this is possible without rewriting large portions of the engine code) Thank you for your amazing feedback, and I apologize that I wasn't able to reply right away! @DevastatioN Right now we're not planning to leverage large amounts of demonstration data to train the agent, the plan is to train the agent to play against itself from scratch (~AlphaZero style). Although this might change in the future. That said, I'd really love to watch some more high-level gameplay (as you said, both pro vs pro and pro vs amateur). This will allow me to understand the task better. So I'd really appreciate if you could send me a few games that you find particularly remarkable. Let's maybe PM about this? Very good questions. I am very excited about incorporating the sound because this is an almost completely unexplored area in AI (combining audio and visual modalities for decision making). Showing that the agent can perform better by using sound would be a very nice result. The plan, in the beginning, is to push it as far as we can from pure visual input, and I hope this bot can reach novice/amateur human level. And from there we will add more domain-specific features, like sound, or maybe something else. Map awareness and timing sense should emerge to a certain degree in the process of training, because we will train the agent on specific map(s), giving it a chance to memorize the layout. On top of that, I have some ideas on how to encourage better temporal and spatial understanding during training, but I want to test these before disclosing. This is also a very interesting research direction, and largely unexplored. Gotcha. Learning more and more about the game mechanics I really started to appreciate this scene. I understand that beating a top human in something like Doom duel will be extremely hard, and therefore this is not necessarily the goal of this project right now. Rather, we want to push it as far as we can, and if we're not able to compete against top humans, we want to identify why, and maybe find out something interesting that current state-of-the-art is missing. This is not about building the best possible bot (because this would involve a lot of hacking, cheating, manual tuning), this is about doing impactful research. What you're suggesting here is called "reward shaping". All these things can definitely help in the learning process but there's always a tradeoff: the more you modify your final objective by adding these auxiliary signals (e.g. positive reinforcement for fast movement), the more the agent will gravitate away from the ultimate goal, which is winning the match. So ideally we want it to just maximize the probability of winning the match, and that's it. After a lot of training, these "auxiliary" objectives should emerge naturally, you can call them instrumental goals, proxies on the way to the terminal goal: winning the match. E.g. when the agent does more damage than the opponent in an exchange, it should increase the internal belief about the probability of winning the match, which will cause positive reinforcement. Ultimately, only experiments will tell just how much reward shaping we will have to use. Maybe some tricks are just extremely hard to learn from pure self-play and we'll have to "teach" the agent. The task of training a general-purpose bot that can beat a human on any map in any scenario seems very hard at this point. I think at the beginning we should focus on deathmatch rather than duel, as DM is less strategic and more reactive, and also we will probably focus on just a few maps. Also, I think duels are more interesting but much more challenging. As I mentioned, the goal is to conduct interesting AI research, not necessarily building the absolute perfect bot. But we like to compare our AI agents to humans, because a skilled human can really demonstate what a truly intelligent agent can pull off, and how far we're from this level.
  12. sun_stealer

    AI research and Doom multiplayer

    Hi, Devastation! Very good and relevant questions. That'd be awesome! If you can provide a few representative examples of high-level gameplay, I'd be very interested! Following the discussion above, I'll be most interested in "vanilla" Doom experience, at least without jumping and vertical aim. In particular, I'd like to see highly skilled deathmatch and duel gameplay on popular classic maps. Sure, I will provide updates, and I think we can use this forum as a communication hub between me and the Doom community. This is working out great so far! The potential training scheme will require billions of game frames of experience, which (at ~35 FPS) is on the order of years or even tens of years of playing Doom for the agent. In real time this will be days and weeks of training on a small GPU cluster. We don't have OpenAI resources to push it further than that, but it'll still be substantial. Keep in mind that this is not a short-term thing, the expected timeline to be on the order of months. I haven't really started working on it yet (finishing some previous stuff), but I already did some setup work and had my first "it's alive!" moment: This is the most primitive baseline algorithms training for just like 30 minutes agains easy bots, so far it only learned to shoot and generally stumble around. Has no clue about healthpacks etc, and doesn't have any memory, etc. But you have to start somewhere. Training is usually is in so-called "episodic setting", in our case I guess one episode will be the entire duel/deatchmatch, and the final objective is to win the match. The agent will try to steer the future towards more favorable states (higher probability of winning, e.g. higher score, better position). The major difference compared to chess is that environment dynamics is considered unknown, we don't use Doom engine to consider different branching probabilities into the future, basically the agent has to figure out how the game works and how to plan directly from pixels by playing the game. Also Doom in partially observable, which means true state is fundamentally unknowable to the agent, therefore memory and awareness are very important. The chess AI, on the other hand, plays each move separately and independently (AKA Markov process). I think from scratch is more interesting, although we might experiment with bootstrapping the agent from some human replays as DeepMind AlphaStar team did. Undecided at this point.
  13. sun_stealer

    AI research and Doom multiplayer

    Yes, I agree with you guys, looks like old-school control style is the way to go. From what I've seen in the tournament videos, it definitely does not look like beating humans will be easy, actually, I expect humans to reign supreme for maybe another year or so. DeepMind is doing similar work with Quake III, but even they couldn't dare to tackle the whole game yet: https://deepmind.com/blog/capture-the-flag-science/ This CTF mode is a very simplified version of Q3, without bunnyhop, without most weapons, etc. Thanks for suggestions @Fonze, I will check Odamex and Doom Explorer now.
  14. sun_stealer

    AI research and Doom multiplayer

    Your video looks much more exciting, maybe because of commentary. Thanks a lot! This is map01, am I right?
  15. sun_stealer

    AI research and Doom multiplayer

    @xvertigox @Decay thank you so much for your replies! I see now, the difference between "oldschool" and "newschool" is very substantial indeed. The existence of freelook and jumping mechanic turns Doom into a completely different game! I'd be interested in using something that people play today, but this might be difficult if Vizdoom does not support it. I will check. Meanwhile, does anyone have footage from recent Doom tournaments (duel, FFA)? Best I could find, is this representative of Doom multiplayer in general?
×