JaxMARL: multi-agent RL environments and algorithms in JAX

Conference item

Benchmarks play an important role in the development of machine learning algorithms, with reinforcement learning (RL) research having been heavily influenced by the available environments. However, RL environments are traditionally run on the CPU, limiting their scalability with typical academic compute. Recent advancements in JAX have enabled the wider use of hardware acceleration to overcome these computational hurdles, enabling massively parallel RL training pipelines and environments. This is particularly useful for multi-agent reinforcement learning (MARL) research. First of all, multiple agents must be considered at each environment step, adding computational burden, and secondly, the sample complexity is increased due to non-stationarity, decentralised partial observability, or other MARL challenges. In this paper, we present JaxMARL, the first open-source code base that combines ease-of-use with GPU enabled efficiency, and supports a large number of commonly used MARL environments as well as popular baseline algorithms. When considering wall clock time, our experiments show that per-run our JAX-based training pipeline is up to 12500x faster than existing approaches. We also introduce and benchmark SMAX, a vectorised, simplified version of the popular StarCraft Multi-Agent Challenge, which removes the need to run the StarCraft II game engine. This not only enables GPU acceleration, but also provides a more flexible MARL environment, unlocking the potential for self-play, meta-learning, and other future applications in MARL. We provide code at https://github.com/flairox/jaxmarl.

Authors: Rutherford, A; Ellis, B; Gallici, M; Cook, J; Lupu, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: Association for Computing Machinery
• Host title: AAMAS '24: Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems
• Pages: 2444-2446
• Publication date: 2024-05-06
• Acceptance date: 2023-12-20
• Event title: 23rd International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024)
• Event location: Auckland, New Zealand
• Event website: https://www.aamas2024-conference.auckland.ac.nz/
• Event start date: 2024-05-06
• Event end date: 2024-05-10
• EISSN: 1558-2914
• ISSN: 1548-8403
• ISBN: 9798400704864
• Language: English