arXiv:2608.09555cs.AI2026-08

通过双向上下文自蒸馏,提升大模型智能体使用外部技能的能力

Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents

论文配图:Bidirectional Context Self-Distillation for Reinforcement Learning of Skill-Based LLM Agents
图 1 · 摘自论文原文
  • 引入双向上下文自蒸馏,从高阶元技能和任务特定技能两视角优化策略
  • 在ALFWorld和WebShop上实现跨模型规模的最优性能,显著提升技能利用率
  • 适合研究大模型智能体决策与技能调用的学者,代码开源可复现

外部自然语言技能为大语言模型(LLM)智能体提供可重用、可编辑的任务指导。然而其效果不仅取决于技能质量,还取决于策略将指导转化为恰当动作的能力。现有方法对提升这种技能利用能力的研究仍不足。实践中,基于技能的智能体通常以任务级奖励为强化学习目标,但监督信号有限,难以捕捉策略使用技能的有效性差异。本文提出BCSD(双向上下文自蒸馏)框架,将自蒸馏与强化学习结合,提升LLM智能体对外部技能的使用效率。不同于以往仅依赖单一优势上下文的方法,BCSD从两个互补的技能上下文视角评估每条轨迹:增强视图引入更高层次的元技能指导,缩减视图去除通用信息以突出任务特定技能。两者在令牌级别产生的信号被融合,用于重标定强化学习的优势值。在ALFWorld和WebShop上的实验表明,BCSD在不同模型规模下均取得最佳整体表现,有效提升技能利用效果。消融实验证实了增强与缩减上下文视图的互补贡献。代码将公开,确保完全可复现。

原文摘要 · Abstract (English)

External natural-language skills provide large language model (LLM) agents with reusable and editable guidance for solving complex tasks. Yet their effectiveness depends not only on skill quality, but also on whether the policy can translate the provided guidance into appropriate actions. However, methods specifically designed to improve this skill-utilization ability remain largely underexplored. In practice, skill-based agents are commonly trained with reinforcement learning objectives centered on task-level rewards, which offer limited supervision and struggle to capture subtle differences in how effectively the policy uses the provided skills. We propose BCSD (Bidirectional Context Self-Distillation), a framework that combines self-distillation with reinforcement learning to train LLM agents to use external skills more effectively. Unlike prior self-distillation methods that rely on a single privileged context, BCSD evaluates each trajectory from two complementary skill-context views. The augmented view introduces higher-level Meta-Skill guidance, while the reduced view prunes general guidance to highlight task-specific skills. Their complementary token-level signals are combined to rescale the RL advantage. Experiments on ALFWorld and WebShop demonstrate that BCSD achieves the strongest overall performance across model scales, enabling agents to utilize external skills more effectively. Ablation studies further verify the complementary contributions of the augmented and reduced context views. Code will be released to ensure full reproducibility.

强化学习大模型智能体技能利用自蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。