Regularized Softmax Deep Multi−Agent Q−Learning

Conference item

Tackling overestimation in Q-learning is an important problem that has been extensively studied in single-agent reinforcement learning, but has received comparatively little attention in the multi-agent setting. In this work, we empirically demonstrate that QMIX, a popular Q-learning algorithm for cooperative multi-agent reinforcement learning (MARL), suffers from a more severe overestimation in practice than previously acknowledged, and is not mitigated by existing approaches. We rectify this with a novel regularization-based update scheme that penalizes large joint action-values that deviate from a baseline and demonstrate its effectiveness in stabilizing learning. Furthermore, we propose to employ a softmax operator, which we efficiently approximate in a novel way in the multi-agent setting, to further reduce the potential overestimation bias. Our approach, Regularized Softmax (RES) Deep Multi-Agent Q-Learning, is general and can be applied to any Q-learning based MARL algorithm. We demonstrate that, when applied to QMIX, RES avoids severe overestimation and significantly improves performance, yielding state-of-the-art results in a variety of cooperative multi-agent tasks, including the challenging StarCraft II micromanagement benchmarks.

Authors: Pan, L; Rashid, T; Peng, B; Huang, L; Whiteson, S

• Publication status: Published
• Peer review status: Reviewed (other)
• Publisher: NeurIPS
• Journal: NeurIPS Proceedings 2021
• Volume: 34
• Pages: 1365-1377
• Publication date: 2022-04-01
• Acceptance date: 2021-11-01
• Event title: 35th Annual Conference on Neural Information Processing Systems (NeurIPS 2021)
• Language: English
• Keywords: FFR