让智能体动态管理外部技能,提升复杂任务表现。
Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
- 通过留一法评估技能贡献,动态调整活跃技能集。
- 在两个基准上平均性能优于基线7.1个百分点。
- 适合研究智能体长期学习与技能演化的工作。
大型语言模型智能体日益依赖外部技能来解决复杂任务,这些技能作为模块化单元扩展其能力边界。现有方法假设外部技能要么持续积累,要么内化为策略,最终导致零技能推理。我们指出该假设过于严格:由于参数容量有限且技能边际贡献不均,最优活跃技能集具有非单调性、任务和阶段依赖性。本文提出SLIM框架,实现动态技能生命周期管理,将活跃外部技能集作为与策略学习联合优化的变量。具体地,通过留一法验证评估每个技能的边际外部贡献,并执行三种生命周期操作:保留高价值技能、淘汰经充分暴露后贡献可忽略的技能、在持续失败时扩充技能库以覆盖缺失能力。实验表明,SLIM在ALFWorld和SearchQA上平均性能优于最佳基线7.1个百分点。结果进一步表明,策略学习与外部技能保留并非互斥:部分技能被内化,部分仍具外部价值,支持SLIM作为更通用的基于技能的智能体强化学习范式。
原文摘要 · Abstract (English)
Large language model agents increasingly rely on external skills to solve complex tasks, where skills act as modular units that extend their capabilities beyond what parametric memory alone supports. Existing methods assume external skills either accumulate as persistent guidance or internalized into the policy, eventually leading to zero-skill inference. We argue this assumption is overly restrictive, since with limited parametric capacity and uneven marginal contribution across skills, the optimal active skill set is non-monotonic, task- and stage-dependent. In this work, we propose SLIM, a framework of dynamic Skill LIfecycle Management for agentic reinforcement learning (RL), which treats the active external skill set as a dynamic optimization variable jointly updated with policy learning. Specifically, SLIM estimates each active skill's marginal external contribution through leave-one-skill-out validation, then applies three lifecycle operations: retaining high-value skills, retiring skills whose contribution becomes negligible after sufficient exposure, and expanding the skill bank when persistent failures reveal missing capability coverage. Experiments show that SLIM outperforms the best baselines by an average of 7.1% points across ALFWorld and SearchQA. Results further indicate that policy learning and external skill retention are not mutually exclusive: some skills are absorbed into the policy, while others continue to provide external value, supporting SLIM as a more general paradigm for skill-based agentic RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。