提升长时序智能体训练效率，让知识蒸馏更关注关键决策阶段。
TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
- 按回合动态调整采样深度，避免无效计算
- 分阶段平衡损失权重，强化深层决策学习
- 适合需要长期规划的对话与任务型智能体
在策略知识蒸馏（OPD）中，学生策略通过匹配教师在自身轨迹上的表现进行训练，为语言智能体训练提供了有前景的框架。然而，其在长时序智能体任务中的应用仍不充分。我们识别出原始方法的两大低效问题：(1) 全程采样常浪费时间在提供弱且噪声大的KL监督的末尾回合；(2) 轨迹级KL目标将大部分损失集中在浅层词元，导致初始行为对齐后深层决策阶段训练不足。为此，我们提出TurnOPD，一种面向长时序智能体的回合级预算策略。该方法包含两个控制器：基于探测的自适应采样深度预算，根据回合统计动态决定采样长度；以及渐进式回合归一化损失预算，逐步将KL权重从词元级转向回合均衡监督。在ALFWorld、WebShop和多跳搜索任务上，使用专用教师模型的实验表明，TurnOPD在相同训练时间下实现更高验证准确率，并显著推进了精度-时间前沿，优于原始OPD。
原文摘要 · Abstract (English)
On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。