让大模型更懂自己推理的进展，提升生成质量
Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
- 用推理进展独立评估替代教师奖励，识别并过滤误导性监督
- 在多个推理任务上，相比标准方法，正确率平均提升6.3%
- 适合需要高质量逻辑推理的文本生成场景
在线策略蒸馏（OPD）通过将学生生成的轨迹与教师提供的密集令牌级监督相结合，成为微调语言模型的有效框架。然而，传统方法假设教师反馈可作为推理进展的可靠代理，对所有教师信号一视同仁。实际上，教师奖励常与真实推理进展相悖：某些推理步骤虽有明显进步，却因偏离教师输出而获得较低奖励。为此，我们提出R2-OPD，构建两个轨迹内推理片段的排名：一个基于教师奖励，另一个基于独立估计的推理进展奖励。当两者不一致时，抑制相应蒸馏奖励，减少与真实推理进展冲突的监督，同时保留有效指导。实验表明，该方法在推理性能上持续优于标准OPD，尤其在复杂逻辑任务中表现显著。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。