用验证器反馈控制教师指导，让小模型学得更准更稳。
Reward-Gated On-Policy Distillation
- 根据验证器反馈决定是否信任教师的逐标记指导
- 在1000步生成长度下比基础方法高2.9分，长序列任务超基线8.2分
- 适合需要高可靠性推理的场景，如代码生成与复杂问题求解
在线策略蒸馏能将强教师模型的推理能力迁移到小型学生模型：学生用自己的策略采样轨迹，教师对实际访问的状态提供密集的逐标记监督。然而，这种监督并不总是可靠：教师可能给看似合理但错误的解分配高概率，或对正确但路径不同的学生解分配低概率。无条件蒸馏可能强化不良模式或抹除有用行为。为此，我们提出奖励门控的在线策略蒸馏（RG-OPD），利用验证器反馈判断何时应信任教师输出。RG-OPD融合稀疏验证器奖励与密集教师输出，在保留逐标记监督的同时过滤误导性信号。在多个推理与编码基准上，RG-OPD生成的蒸馏学生模型更强，优于标准反KL蒸馏和近期的TSD-KD基线。在1000步生成长度下，相比反KL提升2.9分，相比TSD-KD提升4.9分；在长序列设置中，相比未调优学生提升8.2分。代码已公开于https://github.com/UoC-tail/RG-OPD。
原文摘要 · Abstract (English)
On-policy distillation is a powerful way to transfer reasoning ability from a strong teacher to a smaller student: the student samples trajectories from its own policy, and the teacher provides dense token-level supervision on the states the student actually visits. However, this supervision is not always reliable: a teacher can assign high likelihood to plausible but incorrect solutions, or low likelihood to correct student solutions that follow different reasoning paths. Unconditionally distilling the teacher can therefore reinforce bad modes or erase useful student behavior. To address these limitations, we introduce RG-OPD: Reward-Gated On-Policy Distillation that uses verifier feedback to decide when teacher logits should be trusted. RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals. Across reasoning and coding benchmarks, RG-OPD produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline. At 1K generation length, RG-OPD improves over reverse-KL by 2.9 points and over TSD-KD by 4.9 points; in the long-generation setting, it improves over the untuned student by 8.2 points. Our code is available at https://github.com/UoC-tail/RG-OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。