通过预判未来轨迹优化教师指导，提升智能体强化学习的稳定性和效果。
Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
- 在高分歧状态执行短时教师引导，评估其对后续轨迹的影响
- 在ALFWorld等3个任务上平均提升16.6分，优于传统方法
- 适用于不同规模学生模型和教师设置，具有强泛化能力
在线策略蒸馏（OPD）为学生访问的状态提供教师监督，减少训练与推理间的分布差异。但在多轮智能体任务中，学生偏差会随时间累积，逐渐偏离教师指导有效的状态。定量分析表明，高分歧状态蕴含潜在的教师指导机会，但判断指导是否有益需考察其对后续学生轨迹的影响。我们提出FutureBridge-OPD（FTB），在高分歧状态执行短期教师引导，并利用生成的学生延续轨迹评估该引导是否提升了正向蒸馏信号的密度。在ALFWorld、WebShop和ScienceWorld上，以Qwen3-32B为教师、Qwen3-1.7B为学生时，FTB相比原始OPD和TCOD平均分别提升16.6分和7.6分，且在不同学生规模和教师设置下均保持有效性。代码已公开于https://github.com/ChenChiShui/FutureBridge-OPD。
原文摘要 · Abstract (English)
On-policy distillation (OPD) provides teacher supervision on states visited by the student, reducing the distribution gap between training and inference. However, in multi-turn agentic tasks, student deviations may accumulate over time, gradually moving the trajectory away from states where teacher guidance remains effective. Our quantitative analysis further shows that high-disagreement states offer promising opportunities for teacher guidance, but determining whether such guidance is beneficial requires examining its effect on subsequent student trajectories. We propose FutureBridge-OPD (FTB), which executes a short teacher bridge at a high disagreement state and uses the resulting student continuation to assess whether the bridge increases the density of positive distillation signals relative to the teacher. On ALFWorld, WebShop, and ScienceWorld, under the main Qwen3-32B teacher to Qwen3-1.7B student setting, FTB outperforms vanilla OPD and TCOD by an average of 16.6 and 7.6 points, respectively, and remains effective across student scales and teacher settings. Our code is publicly available at https://github.com/ChenChiShui/FutureBridge-OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。