arXiv:2608.05219cs.AI2026-08

让智能体在对话中不被错误指引,只在状态匹配时接收强化指导。

When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

论文配图:When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents
图 1 · 摘自论文原文
  • 按执行状态是否匹配参考轨迹,动态决定是否启用教师指导。
  • 在状态匹配时构建条件化教师上下文,使指导更贴合实际场景。
  • 在两个真实交互任务中显著提升成功率,适合多轮决策系统研究者。

特权的在线策略蒸馏通过同步教师重新评分学生每一轮响应,提供密集监督,其依据是仅训练阶段可用的参考轨迹(如成功路径)。然而在交互环境中,学生的先前动作持续改变执行状态。当学生以不同顺序完成子目标或采取不同行动时,其执行路径可能进入参考轨迹未覆盖的状态,导致参考信息无法为当前状态提供可靠指导,从而引发状态-参考错配。这促使核心目标:提供与学生当前状态兼容的特权引导。本文提出状态匹配路由与情境化自蒸馏(SMRC-SD),显式判断何时且如何由参考轨迹引导在线学生。每轮中,SMRC-SD验证学生当前执行状态是否与参考轨迹中的支持状态匹配;仅在匹配状态下应用蒸馏,过滤掉参考缺乏局部兼容指导的回合。对每个匹配状态,进一步从成功轨迹中构建状态条件化的教师上下文,使监督基于实际达到的状态。在ALFWorld和WebShop上,SMRC-SD均优于无条件的全程成功路径蒸馏。使用Qwen3-1.7B模型,在ALFWorld上任务成功率从0.746提升至0.865,在WebShop上从0.574提升至0.693。受控的路由与上下文消融实验表明,选择局部支持回合和构建状态兼容教师上下文均为性能提升的关键因素。代码已开源于https://github.com/liujunzhuo/SMRC-SD。

原文摘要 · Abstract (English)

Privileged on-policy distillation provides dense supervision for multi-turn agents by allowing a synchronized teacher to re-score the student's response at every turn with access to training-only references, such as successful trajectories. In interactive environments, however, the student's preceding actions continually change the execution state. As the student takes different actions or completes subgoals in a different order, its rollout may reach states not covered by the reference, making the reference an unreliable source of guidance for the state actually reached. Applying privileged distillation indiscriminately therefore creates state--reference mismatch. This mismatch motivates a central objective: providing privileged reference guidance that remains compatible with the student's current execution state. We introduce State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), which explicitly determines when and how a privileged trajectory should guide an on-policy student. At each turn, SMRC-SD verifies whether the student's current execution state matches a supported state along the reference trajectory. Distillation is applied only at matched states, filtering out turns for which the reference lacks locally compatible guidance. For each matched state, SMRC-SD further constructs state-conditioned teacher context from the successful trajectory, grounding supervision in the state actually reached. Across ALFWorld and WebShop, SMRC-SD consistently outperforms unconditional successful full-path distillation. With Qwen3-1.7B, it improves task success from $0.746$ to $0.865$ on ALFWorld and from $0.574$ to $0.693$ on WebShop. Controlled routing and context ablations support both selecting locally supported turns and constructing state-compatible teacher context as contributors to these gains. Code is available at https://github.com/liujunzhuo/SMRC-SD.

多轮对话智能体蒸馏状态匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。