arXiv:2606.19659cs.CL2026-06被引 5

针对多轮对话中教师指导易出错的问题,提出选择性干预框架提升学生模型性能。

SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

论文配图:SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation
图 1 · 摘自论文原文
  • 根据环境反馈决定是否在特定回合进行干预,避免盲目纠错
  • 用教师置信度加权蒸馏,降低不确定指导的影响,提升鲁棒性
  • 适合需要长期推理的智能体训练,尤其适用于复杂任务场景

在线策略蒸馏(OPD)通过让学生模型在自身策略生成的轨迹上训练,缓解智能体训练中的暴露偏差问题。然而,多数现有研究聚焦于单轮场景,而真实大语言模型智能体需在多轮交互中与环境互动。在此情形下,早期错误会改变后续观测并沿轨迹累积,导致标准密集的逐标记级蒸馏变得脆弱:可能过度惩罚语义合理的选择,强化重复动作等局部退化行为,并传播对分布外历史的不可靠教师监督。为此,本文提出SAGE-OPD——一种无需验证器的多轮选择性干预框架。该方法先观察环境反馈,再由教师判断每个学生输出是否应跳过或干预;为缓解错误累积,以教师置信度加权令牌级蒸馏,降低不确定教师分布对异常或模糊历史的影响;最后通过损失归一化保持标准OPD的整体损失尺度,同时保留回合级选择性加权。实验表明,SAGE-OPD在多个智能体任务上持续优于基线,在ALFWorld未见任务中相对标准OPD提升最高达13.3%的成功率。消融实验进一步验证了回合级干预、教师置信度加权与损失归一化的互补增益。结果表明,有效的多轮OPD应保持在线策略特性,但教师监督应仅在必要且可靠的回合中选择性分配。

原文摘要 · Abstract (English)

On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training. However, most OPD studies focus on single-turn settings, while realistic LLM agents interact with environments over multiple turns. In this regime, early errors can alter future observations and compound across the trajectory, and standard dense token-level OPD becomes brittle, as it may over-penalize semantically valid alternatives, reinforce local degeneracies such as repeated actions, and propagate unreliable teacher supervision on off-distribution histories. We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD. Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on. To further address compounding errors, SAGE-OPD weights token-level distillation by teacher confidence, reducing the influence of uncertain teacher distributions on corrupted or ambiguous histories. Finally, SAGE-OPD applies loss normalization to preserve the overall loss scale of standard OPD while retaining selective turn-level weighting. Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD. Ablation studies further demonstrate that turn-level intervention, teacher confidence weighting, and loss normalization provide complementary benefits. Our results suggest that effective multi-turn OPD should remain on-policy, but teacher supervision should be selectively allocated to turns where intervention is necessary and reliable.

多轮对话蒸馏智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。