arXiv:2603.15401cs.SEcs.AI2026-03被引 59

测试49个代码助手技能,发现多数对真实开发帮助有限。

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

  • 用真实项目和需求文档配对技能,做有对照的实测
  • 49个技能中39个没提升通过率,平均只增1.2%
  • 只有少数专业技能有效,错误引导反而让表现下降

Agent技能(推理时注入的结构化过程知识)在软件工程任务中日益普及,但其在端到端开发场景中的实际效用尚不明确。本文提出SWE-Skills-Bench,首个基于需求驱动的基准,隔离评估技能在真实软件工程中的边际价值。该基准将49个公开的SWE技能与固定提交版本的GitHub仓库及包含明确验收标准的需求文档配对,生成约565个任务实例,覆盖六个子领域。我们引入确定性验证框架,将验收标准映射为可执行测试,实现有无技能的受控对比评估。结果表明:技能注入的实际收益远低于快速采纳所暗示的程度——49个技能中有39个未带来通过率提升,平均仅提高1.2%;令牌开销从节省到增加451%,而通过率不变。仅有7个专业技能产生显著效果(最高+30%),另有3个因版本不匹配的指导与项目上下文冲突导致性能下降(最高-10%)。研究提示,技能是窄范围干预,其有效性高度依赖领域适配、抽象层级与上下文一致性。SWE-Skills-Bench为技能设计、选择与部署提供了评估平台。项目地址:https://github.com/GeniusHTX/SWE-Skills-Bench。

原文摘要 · Abstract (English)

Agent skills, structured procedural knowledge packages injected at inference time, are increasingly used to augment LLM agents on software engineering tasks. However, their real utility in end-to-end development settings remains unclear. We present SWE-Skills-Bench, the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering (SWE). It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. We introduce a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Our results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, and the average gain is only +1.2%. Token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance conflicting with project context. These findings suggest that agent skills are a narrow intervention whose utility depends strongly on domain fit, abstraction level, and contextual compatibility. SWE-Skills-Bench provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents. SWE-Skills-Bench is available at https://github.com/GeniusHTX/SWE-Skills-Bench.

代码生成智能体评测技能注入实证研究

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。