arXiv:2608.14036cs.AI2026-08

揭示技能为何有效又为何失效,关键在稳定执行流程而非灌输知识

Demystifying Agent Skills: Why They Work-Until They Don't

论文配图:Demystifying Agent Skills: Why They Work-Until They Don't
图 1 · 摘自论文原文
  • 通过控制实验与轨迹分析,发现技能主要靠稳定动作序列起作用
  • 技能比工作流记忆提升6.06分,65.7%案例依赖流程锚定而非知识注入
  • 检索难度是瓶颈:100个技能时实际命中率降至3.3%,需警惕误干扰

技能作为增强大模型代理推理能力的实用方法,通过结构化知识包提升性能。然而,现有评估多聚焦任务成功率,忽视了更根本的问题:技能何时有效、为何有效、又在何处失效?我们通过跨多个基准测试、代理框架和大模型的受控实验,分离出表征、结果标注、检索难度及跨框架鲁棒性的影响。进一步设计对比研究,结合定量实验与成对轨迹分析,标准化8,135次试验记录,保留238个有效独立标签(来自240条开放式编码记录)。归纳出三类高层级类别与十二种技能使用模式:技能有效时,会将混乱的执行轨迹转化为流程锚点,实现稳定。在匹配比较中,技能比工作流记忆高6.06分。流程锚定解释了65.7%的技能案例,而显式知识注入仅占4.5%,说明技能核心作用是稳定行为而非补充信息。检索是独立瓶颈:当技能池从5增至100时,实际使用精度从29.6%降至3.3%。混淆项虽影响离线识别,但下游成功率仍稳定;精确调用真实答案既非充分也非必要。技能在假设脆弱、上下文不兼容或适应不足时会失败。这些发现推动评估超越整体成功率,指导可自演化的可靠代理构建。

原文摘要 · Abstract (English)

Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \emph{\textbf{When do skills help, why do they work, and where do they fail?}} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.

大模型代理技能机制流程稳定检索瓶颈

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。