让大模型学会主动求助专家，提升推理能力
EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance
- 训练时允许模型自主决定何时向外部专家提问
- 在AIME和AIMO上比自探索强化学习平均高5分
- 适合需要强推理的数学与代码任务场景
大语言模型在基于可验证奖励的强化学习优化后，推理能力显著提升。现有方法多依赖结果反馈来强化内部推理，常导致探索效率低、奖励稀疏。为此，我们提出专家辅助策略优化（EAPO），一种新型强化学习框架，在训练中引入与外部专家的多轮交互，使策略能自适应地判断何时何地寻求帮助，从而获得更丰富的奖励信号和更可靠的推理路径。外部协助最终将专家知识内化到模型中，增强其自身推理能力。评估阶段，模型已充分优化，可独立解题，生成更优推理过程并提高准确性。在AIME 2024/2025和AIMO 2025上，EAPO持续优于专家辅助、专家蒸馏及强化学习基线，平均比自探索强化学习高出5分，并泛化至非数学基准测试，包括HumanEval、HLE、GPQA、MMLU、EvalPlus、HotpotQA和SimpleQA。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently advanced in reasoning when optimized with reinforcement learning (RL) under verifiable rewards. Existing methods primarily rely on outcome-based supervision to strengthen internal LLM reasoning, often leading to inefficient exploration and sparse rewards. To mitigate this issue, we propose Expert-Assisted Policy Optimization (EAPO), a novel RL framework that enhances exploration by incorporating multi-turn interactions with external experts during training. Unlike prior methods, where policies reason in isolation, EAPO incentivizes the policy to adaptively determine when and how to consult experts, yielding richer reward signals and more reliable reasoning trajectories. External assistance ultimately internalizes expert knowledge into the policy model, amplifying the model's inherent reasoning capabilities. During evaluation, the policy model has been well-optimized to solve questions independently, producing improved reasoning paths and more accurate solutions. On AIME 2024/2025 and AIMO 2025, EAPO consistently outperforms expert-assisted, expert-distilled, and RL baselines, averaging a 5-point gain over self-exploration RL, and also generalizes to non-math benchmarks, including HumanEval, HLE, GPQA, MMLU, EvalPlus, HotpotQA, and SimpleQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。