通过首词多样化提升强化学习推理效率，无需修改原有框架。
Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR
- 在推理标记后首词处均匀采样前N个候选词，扩大探索空间。
- 在0.5B-7B模型上，Pass@1、Pass@8、Pass@64均优于基线方法。
- 轻量级改进，适配各类大模型与难度场景的推理训练。
基于可验证奖励的强化学习（RLVR）无需标注轨迹，依赖分组回溯来暴露策略于多种推理路径，并由验证器评分。回溯多样性已成为RLVR的核心瓶颈，现有方法多通过温度、前缀或回溯选择调整来扩展探索。我们发现一个被忽视但结构独特的优化点：推理标记后的首词。策略的首词分布呈现尖锐集中但与正确性解耦的特性，该位置可扩大回溯组覆盖区域，而不改变正确性信号。我们提出REFT（首词多样化回溯探索），作为对RLVR流程的轻量级改进，从策略自身前N个候选中均匀采样首词，并均等分配回溯，其余组件保持不变。在四类基础模型（0.5B-7B）和三种难度条件下，使用多样化回溯训练的REFT，在整体Pass@1、Pass@8、Pass@64上优于DAPO与GRPO基线。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) trains reasoning models without labeled trajectories, relying on grouped rollouts to expose the policy to alternative reasoning paths and a verifier to score them. Rollout diversity has accordingly emerged as a central bottleneck in RLVR, with most existing methods broadening exploration through temperature, prefix, or rollout-selection adjustments. We identify a structurally distinguished but overlooked position for broadening this diversity: the first token after the reasoning marker. The policy's first-token distribution exhibits a sharply peaked yet correctness-decoupled phenomenon, and this first token position can broaden the regions a rollout group covers without altering the correctness signal. We introduce REFT (Rollout Exploration with First-Token Diversification), a light addition to the RLVR pipeline that samples first tokens uniformly from the policy's own top-$N$ candidates and allocates rollouts evenly, leaving every other component unchanged. Trained on the resulting diversified rollouts, REFT improves aggregate Pass@1, Pass@8, and Pass@64 over DAPO and GRPO baselines across four base models (0.5B-7B) and three difficulty regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。