提出一步一指导的蒸馏方法,让模型修复错误路径更完整。
Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

- 用学生生成轨迹的完整步骤做监督,而非逐词修正
- 在ALFWorld上比普通蒸馏提升13.4%成功率
- 适合需要长程纠错的推理与智能体任务
在策略蒸馏(OPD)中,学生模型通过教师对自生成轨迹的输出分布进行对齐,已实现显著性能提升,且所需数据远少于离策略蒸馏。但标准的逐词级OPD仅能提供零散修正,无法构建完整修复路径。为此,我们提出步级在策略蒸馏(SOPD),将监督微调(SFT)的长程修正能力与OPD的在策略优势结合,实现对学生生成轨迹的完整步骤级监督。SOPD在不同步长极限下可退化为SFT或逼近OPD。相比SFT,SOPD的教师响应依赖于学生访问状态,更贴合实际路径;相比OPD,SOPD提供更长视野的修正而非碎片化指导。在推理与智能体任务中,SOPD显著优于传统SFT与OPD。例如,在ALFWorld上,其平均成功率比基础OPD高出13.4个百分点。
原文摘要 · Abstract (English)
On-policy distillation (OPD) aligns a student model with a teacher's logit distribution on student-generated trajectories. This approach has achieved strong empirical gains and can often surpass conventional off-policy distillation with substantially less data. However, standard token-level OPD can provide only fragmented corrections along an erroneous student trajectory and cannot unfold a complete and correct repair path. Motivated by this limitation, we propose \emph{Step-Level On-Policy Distillation} (SOPD), which combines the long-horizon correction of supervised fine-tuning (SFT) with the on-policy advantage of OPD to provide step-level supervision over complete student-generated trajectories. We show that, at different limits of step length, SOPD reduces to SFT or approximates OPD. Compared with SFT, the teacher responses in SOPD are conditioned on student trajectories and therefore align more closely with student-visited states; compared with OPD, SOPD provides longer-horizon corrections rather than fragmented token-level guidance. Across both reasoning and agent tasks, SOPD substantially outperforms conventional SFT and OPD. For example, on ALFWorld, SOPD improves the average success rate by 13.4 points over Vanilla OPD. We hope this work offers a new perspective for future research on distillation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。