用强化学习优化多智能体推理，提升开源模型解题准确率
MarsRL: Advancing Multi-Agent Reasoning System via Reinforcement Learning with Agentic Pipeline Parallelism
- 设计多智能体协同训练框架，分角色奖励减少噪声
- 在Qwen3-30B上将AIME2025准确率从86.5%提至93.3%
- 适合想提升开源模型推理能力的研究者
大语言模型的进展依赖于可验证奖励的强化学习（RLVR）和测试时扩展。然而，语言模型输出长度有限，限制了单次推理中的推理深度。多智能体推理系统通过引入求解器、验证器和修正器等多智能体，迭代优化解决方案，成为有效替代方案。尽管在闭源模型如Gemini 2.5 Pro中表现良好，但其在开源模型上的泛化能力受限于缺乏有效的批评与修正机制。为此，我们提出MarsRL，一种结合智能体流水线并行的新型强化学习框架，用于联合优化系统内所有智能体。MarsRL采用智能体特异性奖励机制以降低奖励噪声，并借鉴流水线训练策略提升长轨迹处理效率。应用于Qwen3-30B-A3B-Thinking-2507模型，其在AIME2025上的准确率从86.5%提升至93.3%，BeyondAIME从64.9%提升至73.8%，甚至超越了Qwen3-235B-A22B-Thinking-2507。结果表明，MarsRL有潜力推动多智能体推理系统发展，并拓展其在多样化推理任务中的应用。
原文摘要 · Abstract (English)
Recent progress in large language models (LLMs) has been propelled by reinforcement learning with verifiable rewards (RLVR) and test-time scaling. However, the limited output length of LLMs constrains the depth of reasoning attainable in a single inference process. Multi-agent reasoning systems offer a promising alternative by employing multiple agents including Solver, Verifier, and Corrector, to iteratively refine solutions. While effective in closed-source models like Gemini 2.5 Pro, they struggle to generalize to open-source models due to insufficient critic and correction capabilities. To address this, we propose MarsRL, a novel reinforcement learning framework with agentic pipeline parallelism, designed to jointly optimize all agents in the system. MarsRL introduces agent-specific reward mechanisms to mitigate reward noise and employs pipeline-inspired training to enhance efficiency in handling long trajectories. Applied to Qwen3-30B-A3B-Thinking-2507, MarsRL improves AIME2025 accuracy from 86.5% to 93.3% and BeyondAIME from 64.9% to 73.8%, even surpassing Qwen3-235B-A22B-Thinking-2507. These findings highlight the potential of MarsRL to advance multi-agent reasoning systems and broaden their applicability across diverse reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。