用多智能体协作提升大模型推理能力，无需人工标注。
MALT: Improving Reasoning with Multi-Agent LLM Training
- 分三步走：生成、验证、精炼，由不同智能体接力完成。
- 在MATH等数据集上比基线模型提升15.66%以上。
- 自动构建训练数据，适合复杂推理任务的研究者。
大语言模型常依赖单一思维链，难以探索多种推理路径或自我修正错误。本文提出MALT（多智能体大模型训练），一种新型后训练策略，将推理过程拆分为生成、验证和精炼三阶段，通过异构智能体的串行流水线实现。在数据生成阶段，每个智能体反复采样形成多智能体搜索树，最终输出根据真实答案评分。我们采用价值迭代反向传播奖励信号，自动构建无需人类或教师模型监督的多智能体训练数据。该离策略方法使各智能体能从正确与错误轨迹中学习，逐步优化端到端推理链。在MATH、GSM8K和CSQA数据集上，MALT相对于相同基线模型分别取得15.66%、7.42%和9.40%的相对提升，推动了多智能体协同训练的发展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often produce answers with a single chain-of-thought, which restricts their ability to explore reasoning paths or self-correct flawed outputs in complex tasks. In this paper, we introduce MALT (Multi-Agent LLM Training), a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps using a sequential pipeline of heterogeneous agents. During data generation, each agent is repeatedly sampled to form a multi-agent search tree, where final outputs are graded against ground-truth data. We then apply value iteration to propagate reward signals back to each role-conditioned model, automatically producing multi-agent post-training data without human or teacher-model supervision. Our off-policy approach allows each agent to specialize by learning from correct and incorrect trajectories, ultimately improving the end-to-end reasoning chain. On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively, making it an important advance towards multi-agent cooperative training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。