arXiv:2604.10674cs.LGcs.AI2026-04被引 45

让AI agent自己总结成功与失败经验,用动态技能指导训练,提升多轮交互效率。

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

  • 用代理自身轨迹生成可变的自然语言技能作为教师监督信号
  • 在AppWorld和Sokoban上相比基础RL提升14.0%~42.1%
  • 适合需要长期规划与自我反思的复杂多轮任务研究者

强化学习(RL)广泛用于训练大模型代理完成多轮交互任务,但稀疏奖励和长时序导致样本效率低下。在策略自蒸馏(OPSD)中,特权教师通过真实答案提供密集的逐令牌监督,但固定监督信息难以覆盖任务中的多样化有效策略,且与RL直接结合常引发训练崩溃。为此,本文提出Skill-SD框架,将代理的完整轨迹提炼为描述成功行为、错误与工作流的紧凑自然语言技能,作为仅教师使用的动态特权信息。学生始终基于原始任务提示行动,通过蒸馏内化指导。为稳定训练,设计重要性加权反KL损失实现梯度修正的逐令牌蒸馏,并动态同步日益优化的学生与教师。在代理基准测试上的实验表明,Skill-SD显著优于标准RL基线,在AppWorld和Sokoban上分别提升14.0%和10.9%(相对于vanilla GRPO),以及42.1%和40.6%(相对于vanilla OPD)。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/

大模型代理自蒸馏强化学习多轮任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。