让教师模型利用学生完整推理轨迹优化教学,提升自蒸馏效果。
PAST: Privileged Adaptation from Complete Student Trajectories for On-Policy Self-Distillation
- 用学生完整推理路径作为额外指导信息,动态调整教师模型。
- 在三个数学推理任务上,平均准确率比基线高5.6个百分点。
- 适合改进需要逐步推理的生成式模型,尤其关注轨迹反馈的场景。
在线策略自蒸馏(OPSD)利用一个特权教师对从自身回放中采样的前缀进行监督。然而,每次回放还揭示了学生响应的完整演化过程及其成败结果,而标准的OPSD并未利用这些学生特有的事后信息。本文提出从学生轨迹中获取特权适应(PAST),将每个完成的学生轨迹视为额外的特权信息用于优化教师,同时保持学生蒸馏前缀不变。PAST 在正确轨迹上保持学生的下一个词分布,并利用失败轨迹在学生邻近正则化下调整教师以实现已验证的成功。我们分析了这种轨迹条件教师可传递给仅依赖前缀的学生的内容。前向KL蒸馏将教师分布投影到给定前缀的条件算术均值,分离出仍为特权的轨迹特异性变化与学生可获取的均值策略迁移。在正确轨迹上,未截断的群体目标也以冻结的学生为理想分布固定点。在三个数学推理基准测试中,PAST 比基线OPSD的宏平均准确率提升5.6个百分点。2×2因子实验表明,完整轨迹访问与教师适应均带来增益;轨迹移除与打乱实验确认,适配后的教师确实利用了匹配的事后上下文。
原文摘要 · Abstract (English)
On-policy self-distillation (OPSD) uses a privileged teacher to supervise a reasoning model on prefixes sampled from its own rollouts. Yet each rollout also reveals how the student's response unfolds and whether it succeeds, student-specific hindsight that standard OPSD does not use to form the teacher. We introduce Privileged Adaptation from Student Trajectories (PAST), which treats each completed student trajectory as additional privileged information for the OPSD teacher while leaving the student's distillation prefixes unchanged. PAST preserves the student's next-token distribution on correct trajectories and uses failed trajectories to adapt the teacher toward verified success under student-proximity regularization. We characterize what such a trajectory-conditioned teacher can transfer to a prefix-only student. Forward-KL distillation projects the teacher distributions to their conditional arithmetic mean given the prefix. This projection separates trajectory-specific variation that remains privileged from the mean policy shift available to the student. For correct trajectories, the unclipped population objective also has the frozen student as an ideal distributional fixed point. Across three mathematical reasoning benchmarks, PAST improves the Avg@12 macro average over Vanilla OPSD by 5.6 percentage points. A $2\times2$ factorial study shows gains from both complete-trajectory access and teacher adaptation, while trajectory removal and shuffling confirm that the adapted teacher uses the matching hindsight context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。