控制推理长度可大幅提升策略蒸馏效率，无需全程完整推演。
Are Full Rollouts Necessary for On-Policy Distillation?
- 渐进式或截断式控制推理长度，避免早期低质量反馈。
- 截断蒸馏仅用10%长度达到原方法性能，提速3倍。
- 适合资源受限场景下的高效强化学习训练。
在线策略蒸馏（OPD）通过学生生成轨迹中的密集教师反馈替代固定教师轨迹，成为一种有前景的后训练范式。然而标准OPD通常在训练中生成完整轨迹，计算开销大，且早期训练时后期轨迹的教师反馈可能不可靠。本文识别出推理长度是影响OPD效率的关键瓶颈。与需最终答案奖励的可验证奖励强化学习不同，OPD无需最终奖励即可提供学习信号，因此全轨迹并非必需。基于此，提出两种简单长度控制策略：渐进式OPD（POPD）逐步扩展推理长度，截断式OPD（TOPD）永久使用可靠截断轨迹进行蒸馏。在数学推理任务上的实验表明，POPD使训练效率提升最高达3倍；TOPD仅用10%的推理长度即达到原始性能，显著降低墙钟时间和内存占用。结果表明，控制推理长度为实现更高效的OPD提供了简单而实用的路径。
原文摘要 · Abstract (English)
On-policy distillation (OPD) provides dense teacher feedback along student-generated rollouts rather than fixed teacher traces and has emerged as a promising post-training paradigm. However, standard OPD typically generates full rollouts during training, which is computationally expensive and may expose the student to unreliable teacher feedback at late rollout positions, especially during early training. We identify the rollout horizon as a key bottleneck in OPD that substantially impacts training efficiency. Unlike Reinforcement Learning with Verifiable Rewards (RLVR), OPD does not require a final answer reward to provide learning signals. Therefore, full rollouts may not always be necessary for OPD. Motivated by this insight, we propose two simple horizon-control strategies: Progressive OPD (POPD), which gradually expands the rollout horizon during training, and Truncated OPD (TOPD), which permanently performs distillation on reliable truncated rollouts. Experiments on mathematical reasoning show that POPD improves the training efficiency of OPD by up to 3$\times$, while TOPD matches OPD performance using only 10\% of the rollout horizon, leading to substantial wall-clock and memory reductions. These results demonstrate that controlling the rollout horizon offers a simple and practical path to more efficient OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。