解决自洽推理中教师指导信号衰减问题，提升长序列生成质量
Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation
- 用前瞻分组奖励机制评估候选词的未来教师置信度，动态增强监督信号
- 7B模型在6个基准上平均提升2.57点，长序列任务最高达4.92点增益
- 适合需要长程推理的数学与代码生成场景，兼顾效率与效果
在策略内知识蒸馏中，学生模型通过自身生成轨迹和教师提供的逐标记反馈学习推理能力。然而我们发现一个关键瓶颈——监督保真度衰减（SFD）：随着学生生成前缀长度增加，教师对下一个标记的分布变得越来越不确定且区分度下降。这导致反KL蒸馏中的教师修正信号减弱，引发学生偏差在长推理链中累积。为此，我们提出前瞻分组奖励（Lookahead Group Reward）。基于下一时刻教师置信度反映未来监督强度的洞察，该方法评估学生顶K候选词在后续步骤所引发的教师置信度，并进行组归一化奖励。为保持计算效率，进一步设计了熵触发的树注意力机制。在六个数学与代码基准上，该方法使7B学生模型的mean@8提升2.57点，长序列任务增益更高，在AIME-26（39k token）上达到+4.92点。
原文摘要 · Abstract (English)
On-policy distillation transfers reasoning capabilities by training a student model on its own generated trajectories using token-level feedback from a teacher. However, we identify a critical bottleneck, \textbf{Supervision Fidelity Decay (SFD)}: as student-generated prefixes lengthen, the teacher's next-token distribution becomes less confident and less discriminative. Consequently, the teacher-dependent corrective signal in reverse-KL distillation weakens, causing student drift to compound across long reasoning chains. To mitigate SFD, we introduce \textbf{Lookahead Group Reward (\ours{})}. Building on the insight that next-step teacher confidence reflects the discriminative strength of future reverse-KL supervision, \ours{} evaluates the student's top-K candidate tokens by the teacher confidence they induce at the subsequent step and assigns a group-normalized reward. To maintain computational efficiency, we further design an entropy-triggered tree-attention mechanism. Across six math and code benchmarks, \ours{} improves mean@8 by \textbf{2.57} points over OPD for a 7B student, with gains increasing in longer-generation and reaching +\textbf{4.92} points on AIME-26 at 39k tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。