arXiv:2607.27937cs.AI2026-07

让大模型代理通过验证结果来学习更好行为,提升任务成功率。

From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents

论文配图:From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
图 1 · 摘自论文原文
  • 用环境结果验证教师决策,避免错误引导。
  • 在失败路径上构建前缀树,只保留成功延续部分。
  • 适合想提升大模型推理能力的研究者和开发者。

当前大模型代理研究正从外部能力获取转向内部能力固化,使代理无需推理时检索即可保留实用技能。现有基于策略的自蒸馏(OPSD)方法通常通过评分学生生成轨迹中的动作进行监督,但存在两个局限:教师偏好未经过环境结果验证,且动作级评分未能充分利用学生与教师轨迹及行为关系的信息。为此,本文提出结果验证的对比自蒸馏(OVCSD)。OVCSD将失败的学生轨迹组织成前缀树,从学生到达的状态动态调用技能条件教师,并仅保留经结果验证的成功延续。随后在首次状态对齐分歧点处执行局部对比学习,并蒸馏分歧后的教师后缀以传递完成行为。在ALFWorld和WebShop三个模型规模上的实验表明,OVCSD持续优于无技能强化学习和现有自蒸馏基线,在ALFWorld和WebShop上分别实现最高29.7%和5.4%的绝对成功率提升,训练期间额外特权交互不足3%。

原文摘要 · Abstract (English)

Recent work on LLM agents is shifting from external capability elicitation to capability internalization, enabling agents to retain useful skills without retrieval at inference time. On-policy self-distillation (OPSD) offers a promising direction, but many existing methods typically supervise students by scoring actions along student-generated trajectories. Such supervision has two limitations: teacher preferences are not validated by environment outcomes, and action-level scores underuse information from student rollouts, teacher rollouts, and their behavioral relationship. We therefore advocate outcome-verified teacher supervision and comparative learning over teacher-student trajectories. Based on this view, we propose Outcome-Verified Comparative Self-Distillation (OVCSD). OVCSD organizes failed student rollouts into a prefix tree, adaptively invokes a skill-conditioned teacher from student-reached states, and retains only outcome-verified successful continuations. It then applies localized comparative learning at the first state-aligned divergence and distills the post-divergence teacher suffix to transfer completion behavior. Experiments on ALFWorld and WebShop across three model scales show that OVCSD consistently outperforms skill-free RL and existing self-distillation baselines, achieving up to 29.7 and 5.4 absolute success-rate gains over the strongest baselines on ALFWorld and WebShop, respectively, while adding less than 3% privileged interaction during training.

大模型代理自蒸馏强化学习任务规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。