扩增技能库反而让大模型代理表现下降,主因是选错技能而非上下文变复杂。
More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries

- 分解性能下降为选错技能和上下文过载两类影响
- 技能库从少量增至202项时,成功率最高降21%
- 选错技能是主因,上下文影响可忽略
技能库使大模型代理能按需加载特定任务指令,让非专家用户通过自然语言完成专业任务。然而,随着技能库规模扩大,性能显著下降——从少量有效技能扩展至202项技能库时,成功率最高降低21%。本文将性能下降定义为加载已知有效技能库与完整技能库之间的通过率差异,并基于代理在轨迹中调用的技能,分解该下降为两种效应:技能遮蔽(代理在库增大时更易选择错误技能)和上下文开销(即使选择正确,上下文过大会降低执行效果)。我们推导了两者的上限,实证估计显示技能遮蔽效应随库规模增长且是性能下降的主要原因,而上下文开销效应极小,近乎为零。这一不对称性表明,扩大技能库时,技能选择失败是主要瓶颈,而非上下文膨胀。
原文摘要 · Abstract (English)
Skill libraries allow LLM agents to load task-specific instructions on demand, letting non-expert users solve domain-specific tasks through natural language without knowing which skills exist or how they work. However, performance degrades as libraries grow -- by up to 21\% when scaling from a small set of helpful skills to a 202-skill library. In this work, we formulate this performance degradation as the pass rate drop between loading a library of known-helpful skills and the full library. Moreover, we propose to decompose the pass rate drop by conditioning on the skill(s) invocation -- which skills the agent selects during a trajectory -- into two effects: \emph{skill shadowing}, where the agent selects wrong skills more often as the library expands, and \emph{context overhead}, where the enlarged context degrades execution even when selection is correct. We derive upper bounds on both effects to characterize their magnitudes of impacts to the pass rate drop. Our empirical estimates of the effects and their upper bounds both show that the \emph{skill shadowing} effect grows with library size and significantly contributes to the performance degradation, whereas the \emph{context overhead} effect remains small and indistinguishable from zero. This observed asymmetry establishes that the skill selection failure, not the enlarged context, is the primary bottleneck when expanding the skill libraries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。