发现强化学习蒸馏中后期令牌监督失效，提出加权优化方法提升训练效率
On the Position Bias of On-Policy Distillation
- 按累积偏差动态加权学生轨迹中的令牌，优先强化早期高质量监督
- 仅用前30%令牌即达全量效果，后30%令牌几乎无法学习
- 新方法在同规模和跨规模设置下均提速显著，最高提升6.9分
基于策略蒸馏（OPD）通过教师模型提供密集的逐标记监督来提升强化学习效率。标准KL目标对所有标记均匀加权，但研究发现：随着学生轨迹增长，其分布逐渐偏离教师，导致后期标记监督质量下降。实验表明，仅使用前30%标记的OPD表现接近全量使用，而仅用后30%几乎无法学习。本文从约束优化视角剖析该位置偏差问题，提出重要性加权的OPD（IW-OPD），根据学生与教师分布的累积差异动态调整各标记权重，自然提升早期标记权重、降低偏差大的后期标记权重。结果表明，IW-OPD收敛更快，学习效率更高，在同规模与跨规模设置下均优于标准OPD，AIME-2025上性能提升最高达6.9点。
原文摘要 · Abstract (English)
On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher's distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student's and teacher's distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。