Networked communication for decentralised agents in mean-field games and control

Thesis

The mean-field framework analyses the limiting case when an infinite number of agents have common reward and transitions functions, and interact with each other not on a per-agent basis, but instead through a distribution over the other agents’ states (the mean field). The framework can provide approximate solutions for the equivalent problems involving large but finite populations, which can be much harder to solve in themselves. The framework can therefore be used to tackle computational scalability issues facing other paradigms such as multi-agent reinforcement learning (MARL), with applications considered in a wide variety of real-world problems.

However the framework has traditionally been largely theoretical, and classical approaches usually involve assumptions or algorithmic settings that might be restrictive when applied to very large populations of agents deployed in the real world. In particular, centralised methods have typically been used, despite the fact that a single central coordinator is arguably a strong and undesirable assumption for large populations in the real world. On the other hand, entirely independent agents often learn impractically slowly.

We therefore introduce decentralised, networked communication to the framework and show that it mitigates the drawbacks of both baseline architectures. We first introduce it to the non-cooperative mean-field game (MFG) to compare with existing theoretical sample guarantees for the other architectures, before moving from tabular to function approximation settings, and then ultimately to the cooperative mean-field control (MFC) problem. We similarly build extensive theory for these latter settings, proving that our communication scheme actually permits faster learning than both the independent and the centralised alternatives under certain conditions. In all settings we demonstrate the benefits to learning speed experimentally, and we also provide additional studies showing that our networked populations are more robust than the other architectures to unpredicted shocks that may occur in real-world, non-idealised settings.

Authors: Benjamin, P

• DOI: 10.5287/ora-kknpn1vvr
• Type of award: DPhil
• Level of award: Doctoral
• Awarding institution: University of Oxford
• Language: English
• Keywords: mean-field control; reinforcement learning; networked communication; mean-field games
• Subjects: Wireless communication systems; Reinforcement learning; Mean field theory; Machine learning; Game theory