让大模型在关键节点获得专家指导，提升强化学习中的探索效率与多样性。
Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
- 仅在关键决策点提供专家指引，避免全程模仿
- 实验显示探索质量更高，整体性能显著优于基线
- 适合需要高质量推理的LLM强化学习场景
基于可验证奖励的强化学习（RLVR）已成为提升大型语言模型（LLMs）推理能力的主流方法。然而，其效果高度依赖基础模型的能力，原因在于模型需具备高质量探索能力，包括有效性和多样性。现有方法通过模仿专家轨迹来改进，虽提升了有效性，却忽视了多样性。为此，我们提出：专家只需在关键决策点提供引导，而非全程介入。基于此，我们设计了MENTOR框架——一种在分词级别优化推理的混合策略专家导航方法，仅在关键节点提供指导，实现高效且多样化的探索。大量实验表明，该方法使模型捕捉专家策略的本质，而非表面模仿，从而实现高质量探索并取得更优整体表现。代码已公开。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has become a widely adopted technique for enhancing the reasoning ability of Large Language Models (LLMs). However, the effectiveness of RLVR strongly depends on the capability of base models. This issue arises because it requires the model to have sufficient capability to perform high-quality exploration, which involves both effectiveness and diversity. Unfortunately, existing methods address this issue by imitating expert trajectories, which improve effectiveness but neglect diversity. To address this, we argue that the expert only needs to provide guidance only at critical decision points rather than the entire reasoning path. Based on this insight, we propose MENTOR: Mixed-policy Expert Navigation for Token-level Optimization of Reasoning, a framework that provides expert guidance only at critical decision points to perform effective and diverse exploration in RLVR. Extensive experiments show that MENTOR enables models capture the essence of expert strategies rather than surface imitation, thereby performing high-quality exploration and achieving superior overall performance. Our code is available online.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。