针对多轮对话中教师指导易出错的问题,提出选择性干预框架提升学生模型性能。
SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

- 根据环境反馈决定是否在特定回合进行干预,避免盲目纠错
- 用教师置信度加权蒸馏,降低不确定指导的影响,提升鲁棒性
- 适合需要长期推理的智能体训练,尤其适用于复杂任务场景
在线策略蒸馏(OPD)通过让学生模型在自身策略生成的轨迹上训练,缓解智能体训练中的暴露偏差问题。然而,多数现有研究聚焦于单轮场景,而真实大语言模型智能体需在多轮交互中与环境互动。在此情形下,早期错误会改变后续观测并沿轨迹累积,导致标准密集的逐标记级蒸馏变得脆弱:可能过度惩罚语义合理的选择,强化重复动作等局部退化行为,并传播对分布外历史的不可靠教师监督。为此,本文提出SAGE-OPD——一种无需验证器的多轮选择性干预框架。该方法先观察环境反馈,再由教师判断每个学生输出是否应跳过或干预;为缓解错误累积,以教师置信度加权令牌级蒸馏,降低不确定教师分布对异常或模糊历史的影响;最后通过损失归一化保持标准OPD的整体损失尺度,同时保留回合级选择性加权。实验表明,SAGE-OPD在多个智能体任务上持续优于基线,在ALFWorld未见任务中相对标准OPD提升最高达13.3%的成功率。消融实验进一步验证了回合级干预、教师置信度加权与损失归一化的互补增益。结果表明,有效的多轮OPD应保持在线策略特性,但教师监督应仅在必要且可靠的回合中选择性分配。
原文摘要 · Abstract (English)
On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training. However, most OPD studies focus on single-turn settings, while realistic LLM agents interact with environments over multiple turns. In this regime, early errors can alter future observations and compound across the trajectory, and standard dense token-level OPD becomes brittle, as it may over-penalize semantically valid alternatives, reinforce local degeneracies such as repeated actions, and propagate unreliable teacher supervision on off-distribution histories. We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD. Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on. To further address compounding errors, SAGE-OPD weights token-level distillation by teacher confidence, reducing the influence of uncertain teacher distributions on corrupted or ambiguous histories. Finally, SAGE-OPD applies loss normalization to preserve the overall loss scale of standard OPD while retaining selective turn-level weighting. Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD. Ablation studies further demonstrate that turn-level intervention, teacher confidence weighting, and loss normalization provide complementary benefits. Our results suggest that effective multi-turn OPD should remain on-policy, but teacher supervision should be selectively allocated to turns where intervention is necessary and reliable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。