通过位置加权提升推理中教师令牌的可靠性，显著改进自蒸馏效果。
When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning
- 基于轨迹位置设计权重，动态评估教师生成词元的可靠性。
- 在AIME 2024和2025上分别提升1.0和1.1分，效果稳定。
- 无需额外计算教师，适合大模型推理优化场景。
在策略自蒸馏（OPSD）中，学生模型使用自身采样结果与特权教师进行训练，但传统方法对所有生成词元施加相同权重，隐含假设教师输出在每个前缀处均同等可靠。现有基于熵的方法虽尝试调整监督强度，但在推理任务中高熵可能表示无效不确定性或有益解多样性，难以判断可靠性。为此，本文提出分支可行性诊断：记录教师提示下的多路径候选词元，强制学生模板继续生成并检验是否恢复正确答案。在Qwen3-4B上，序列内位置得分是最佳预测指标，其受试者工作特征曲线下面积（AUROC）达0.83，局部不确定性最高仅0.57。据此，我们提出位置加权自蒸馏（PW-OPSD），在保持原有学生轨迹、教师输入及裁剪前向KL目标的前提下，逐步增加位置权重。综合实验表明，该方法在不同随机种子下使AIME 2024和AIME 2025 Avg@12分别提升+1.0和+1.1；在两个更大规模、跨家族模型（DeepSeek-R1-Distill-Llama-8B 和 Olmo-3-7B-Think）上的泛化测试也显示一致的平均性能提升。结果表明，推理中的教师可靠性具有轨迹结构特征，可被有效利用而无需额外教师计算。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) trains a student on its own rollouts using a privileged teacher, but its standard objective weights all generated tokens equally, implicitly treating the privileged teacher target as equally reliable at every student-visited prefix. Existing entropy-based OPD methods relax this uniformity by modulating token-level supervision with teacher entropy, but high teacher entropy in reasoning has an ambiguous reliability meaning: it can reflect either non-viable uncertainty or benign solution diversity. To identify this phenomenon, we introduce a branch-viability diagnostic. Specifically, we record next-token alternatives from the privileged-answer teacher prompt, force each alternative after the student prompt plus its on-policy spine prefix, and test whether the resulting student-template continuation recovers the correct answer. On Qwen3-4B, we find that an oriented within-sequence position score is the strongest tested predictor of teacher-token reliability, reaching an area-under-ROC-curve (AUROC) of 0.83; local uncertainty scores are at most 0.57. Motivated by this trajectory-level structure, we propose Position-Weighted On-Policy Self-Distillation (PW-OPSD), which applies an increasing position weight while keeping the same student rollout, privileged teacher pass, and clipped forward-KL target as OPSD. In our comprehensive evaluations with different random seeds, the diagnostic-derived PW-OPSD improves AIME 2024 and AIME 2025 Avg@12 by +1.0 and +1.1 points, and a generalization evaluation on two larger-scale models from different families, DeepSeek-R1-Distill-Llama-8B and Olmo-3-7B-Think, also demonstrates consistent aggregate Avg@12 improvements. These results show that teacher-token reliability in reasoning distillation is trajectory-structured and can be utilized without additional teacher computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。