提出时间课程蒸馏框架,解决多轮智能体训练中误差累积问题。
TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agents

- 设计渐进式训练课程,逐步增加学生模型看到的轨迹深度。
- 在三个基准上提升性能最高达18点,且能超越教师模型表现。
- 适合需要稳定多轮推理能力的轻量级智能体部署场景。
在策略蒸馏(OPD)中,尽管其在静态单轮任务上表现优异,但在多轮智能体设置下的行为仍不明确。本文发现原始OPD存在轨迹级KL不稳定问题:随着成功率下降,KL散度上升,即使收敛后仍保持高位,导致训练不稳定。根源在于跨轮次错误累积,使学生超出教师的有效支持范围,监督信号失效。为此,我们提出TCOD(时间课程策略蒸馏),通过渐进式扩展暴露给学生的轨迹深度,实现从短到长的课程调度。在四个学生-教师对、三个多轮智能体基准(ALFWorld、WebShop、ScienceWorld)上的实验表明,TCOD有效缓解了KL增长,提升了训练稳定性,性能最高比原始OPD提升18点。进一步评估显示,TCOD甚至可超越教师表现,并泛化至教师无法完成的任务。代码已开源。
原文摘要 · Abstract (English)
On-policy distillation (OPD) has shown strong potential for transferring reasoning ability from frontier or domain-specific models to smaller students. While effective on static single-turn tasks, its behavior in multi-turn agent settings remains underexplored. In this work, we identify a key limitation of vanilla OPD in such settings, which we term Trajectory-Level KL Instability. Specifically, we observe that KL divergence increases together with a drop in success rate, and even after convergence, the KL remains high, leading to unstable training. This instability arises from inter-turn error compounding: as errors accumulate, the student is driven beyond the teacher's effective support, rendering the supervision signal unreliable. To address this, we propose TCOD (Temporal Curriculum On-Policy Distillation), a simple yet effective framework that controls the trajectory depth exposed to the student and progressively expands it from short to long with a curriculum schedule. Experimental results across four student-teacher pairs on three multi-turn agent benchmarks (ALFWorld, WebShop, ScienceWorld) show that TCOD mitigates KL escalation and enhances KL stability throughout training, improving agent performance by up to 18 points over vanilla OPD. Further evaluations show that TCOD can even surpass the teacher's performance and generalize to tasks on which the teacher fails. Our code is available at https://github.com/kokolerk/TCOD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。