MALT: improving reasoning with multi-agent LLM training

Conference item

Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively, making it an important advance towards multi-agent cooperative training.

Authors: Motwani, SR; Smith, C; Das, RJ; Rafailov, R; Laptev, I

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: OpenReview
• Article number: 466
• Publication date: 2025-07-08
• Acceptance date: 2025-07-07
• Event title: 2nd Conference on Language Modeling (COLM 2025)
• Event location: Montreal, Canada
• Event website: https://colmweb.org/index.html
• Event start date: 2025-10-07
• Event end date: 2025-10-10
• Language: English