通过接力式引导纠正学生模型的错误推理方向，提升生成质量与效率。
Pass the Baton: Trajectory-Relayed On-Policy Distillation
- 在学生出错时由教师短暂接管生成，形成接力轨迹进行优化。
- 平均提升5.73%性能，训练长度减少超50%，0.6B和1.7B模型均有效。
- 适合追求高效高质推理的轻量级模型部署场景。
在策略蒸馏中，学生模型一旦进入错误推理路径，后续生成会持续偏离，导致无效监督和算力浪费。本文发现教师与学生在失败前缀上的行为不对称：教师倾向于修正方向，而学生则延续错误。基于此，提出无标签接力式策略蒸馏（Relay-OPD），在检测到错误点时让教师短暂接管生成教师段，之后学生恢复生成并优化整个轨迹。有限的接力预算聚焦于早期关键位置，同时保持对学生策略的近似。在八个数学推理基准上，使用Qwen3-4B-Instruct-2507作为教师、Qwen3-0.6B/1.7B-Non-Thinking作为学生，Relay-OPD在所有任务中达到最佳或第二佳表现，1.7B模型平均优于标准OPD 5.73%、优于最强基线FastOPD 1.49%，0.6B模型同样显著提升，训练轨迹长度减少超过50%。
原文摘要 · Abstract (English)
On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。