An experiment in training cooperative RL agents in Unity and advantages of using RL in game development and testing.
Reinforcement Learning (RL) has gained significant traction in the realms of game development and testing in recent years. The Unity ML-Agents package, a widely-used tool for implementing reinforcement learning in game development, has released version 2.0. This update enables the training of complex cooperative behaviors, allowing developers to design and train agents capable of collaborating to achieve a shared objective.
In this blog, we will highlight an experiment that focuses on training cooperative RL agents within a modified version of the Push Block game environment. This example, included in the ML-Agents package, demonstrates the training of cooperative agents. Additionally, we will discuss the tweak made to the reward function to foster collaboration between agents and briefly touch on the advantages of utilizing RL in game development and testing.
Figure 1. The image on the left is the original Push Block game environment, while the image on the right is the modified version used in our experiment. Two major changes have been made: the heavy blocks now have a different shape, and the agents are now tasked with pushing them towards the light blue walls.
For our experiment, the game environment used to train the RL agents was based on the Push Block game, albeit with a few modifications. To provide a challenge for the cooperative agents, we altered the heavy blocks, making them tall and slender rather than wide. In order to maneuver these blocks effectively, agents must adopt strategic positions, such as sharing space to push the blocks simultaneously or one agent pushing from behind another agent who is directly engaged with the block.
Initially, when employing the legacy reward function from the original Push Block game, the agents struggled to learn how to push the tall blocks towards the walls. The original reward function basically assigned +1 for every small block and +2 for every heavy tall block that reached the goal. However, while moving around to explore pushing the blocks, the agents rarely managed to push any heavy block to the goal by chance, resulting in no relevant rewards during the training for the heavy blocks.
To motivate the agents to continue pushing the tall heavy blocks until they reached the walls, we refined the reward function by adding a small reward (+0.001) each time a heavy block moved further from its starting position. This modification encouraged the agents to push the tall heavy blocks farther during the new training, eventually receiving a substantial reward when achieving the goal. Consequently, after several hours of training, the agents consistently pushed the tall heavy blocks to their intended destinations.
Figure 2. The gray line is the group cumulative reward during training for the improved reward function, while the blue line is from using the legacy reward function.
For more details about the experiment, you can watch the video below:
Reinforcement learning has found applications in numerous game development and testing contexts, including game AI, game balancing, and game testing. For instance, RL has been employed to train agents capable of playing games like Chess, Go, and Poker at superhuman levels. It has also been used for balancing game mechanics, such as adjusting difficulty levels based on player performance, and testing game scenarios through simulating various player actions and outcomes.
Integrating RL, especially Cooperative RL, into game testing offers several practical benefits and advantages for game studios:
1. Enhanced test coverage: Cooperative RL can explore a broader range of game states and scenarios by simulating multiple agents working together or competing against each other. This increased coverage helps identify issues and edge cases that might have been missed through traditional testing methods.
2. Dynamic difficulty adjustment: RL can be used to develop adaptive AI systems that automatically adjust their strategies and difficulty levels in response to player performance, ensuring a more engaging and tailored gaming experience for players with varying skill levels.
3. Improved game balance: By using RL agents to explore different gameplay strategies and interactions, game studios can gain valuable insights into how various game elements interact and identify potential imbalances or unintended dominant strategies. This knowledge allows developers to fine-tune and balance game mechanics more effectively.
4. Reduced development time: RL can automate and accelerate the testing process, freeing up development resources and reducing the time spent on manual testing. As RL agents rapidly iterate and learn from their experiences, they can quickly identify issues that might take human testers significantly longer to discover.
5. Realistic and challenging AI: RL can be employed to create AI agents that exhibit more human-like behavior and adapt to changing game conditions, providing players with a more challenging and immersive gaming experience.
6. Uncovering novel strategies: RL agents can discover unexpected or innovative strategies, potentially enhancing the gameplay experience and inspiring developers to design new game features or mechanics.
7. Continuous improvement: Unlike scripted agents, RL agents can continually learn and improve their performance over time, leading to more robust and reliable game testing outcomes.
While it remains challenging for RL agents to fully replace scripted agents in large games with intricate scenes and scenarios, using RL agents, particularly those trained using cooperative approaches, serves as an excellent method to complement scripted agents and unlock numerous benefits for game studios.
As technology continues to evolve, we can anticipate even more inventive applications of RL within the gaming industry. With tools like the Unity ML-Agents package, game developers can craft increasingly complex and sophisticated AI agents that collaborate to accomplish objectives and interact with their surroundings in more authentic ways.
If you are interested in Reinforcement Learning and testing and what it can do for you, contact us.
References:
1. https://blog.unity.com/engine-platform/ml-agents-v20-release-now-supports-training-complex-cooperative-behaviors
2. Augmenting Automated Game Testing with Deep Reinforcement Learning (https://arxiv.org/abs/2103.15819)
