arXiv:2608.11888cs.AI2026-08被引 1

研究发现大模型技能反而导致任务失败和效率下降,揭示了技能使用中的隐藏风险。

Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents

  • 通过对比有无技能的执行结果,定位具体技能引发的问题
  • 发现307次失败,其中125次功能错误,182次效率降低
  • 技能常让验证和实现变成强制流程,增加冗余开销

Agent技能是扩展大模型代理可复用引导能力的主流方式,影响规划、工具调用、问题解决和验证等环节。已有研究对技能效果评价不一:部分提升成功率,部分无效甚至降低成功率并增加消耗。本文在SkillsBench和SWE-Skills-Bench上构建差分分析框架,通过对比带技能与无技能或语义匹配技能的运行,识别出307项技能引发的失败,包括125项功能失败和182项效率退化。我们还开发了SkillTriage工具,实现案例标准化、差异证据提取与诊断报告生成。主要发现:(1) 功能失败多由看似相关但误导性的技能引起,导致遗漏或错误实现必要步骤;(2) 效率下降不能仅由提示长度解释;(3) 过度流程中,过度验证(67例)和重型实现管道(30例)是主要来源,表明技能常将验证清单和构建流程变为强制任务。基于此,提出更安全、更成本敏感的技能复用研究方向与工具改进方案。

原文摘要 · Abstract (English)

Agent skills are the de facto mechanism for extending LLM agents with reusable guidance. A skill can shape the agent's task execution, including planning, tool use, problem-solving, and validation. Prior work reported mixed results of agent skills: some skills improve task success rates, while others have no effect, increase token use and execution time, and even reduce success rates. This paper presents a comprehensive analysis of skill-induced agent failures by attributing task failures and cost regressions to specific loaded skills. We introduce a differential analysis framework that attributes a failure or regression to a skill by comparing a target skill-guided run against a no-skill or semantically matched skill reference run that solves the same task, or solves it more cheaply. We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions. We also build SkillTriage, a taxonomy-guided attribution tool that normalizes paired cases, extracts differential evidence, and produces triage reports. Our major findings include: (1) Skill induced functional failures are rarely caused by obviously irrelevant skills; instead, seemingly relevant skills often make the agent incorrectly implement or omit task-required implementation elements. (2) Skill-induced efficiency regressions are not explained by prompt length alone. (3) The largest sources within Excessive Procedure are excessive verification and heavy implementation pipelines, contributing 67 and 30 cases, respectively. This shows that skills often turn validation checklists and construction recipes into mandatory work. Based on our findings, we propose research topics and tooling improvements for safer and more cost-aware skill reuse.

大模型代理技能失效效率优化自动化验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。