测试31个网页开发技能，发现多数反而降低代码生成效果。
Signal or Noise? A Benchmark Study of Agent Skills in Web Development
- 通过控制提示长度对比，分离出技能真实影响
- 技能注入使成功率下降1.3%~4.2%，令牌成本上升72%~394%
- 仅17%~36%的组合真正受益，需按模型项目逐个评估
Agent Skills 是可复用的程序模块，被注入编码代理会话以表达框架规范、反模式和工具。然而，每个技能都会增加每次查询的提示长度。有效的技能基准必须判断任务是否真能被解决，以及该技能是否值得注入。本文提出 WebDev-Skills-Bench，对31个公开的WebDev技能在50个Web-Bench项目和1,000个有序任务上进行受控实证研究。基准比较四种匹配条件，包括长度匹配的无关对照组和留一法组件消融。为隔离技能效应与提示长度伪影，仅将 SKILL.md 放入提示，辅助文件挂载至代理工作区。在四个模型上，目标技能注入使平均 Pass@2 下降1.3%至4.2%，降低任务完成深度，并使令牌成本增加72%至394%，仅在17%至36%的技能-项目组合中带来增益。长度匹配对照揭示两种失败模式：部分模型存在长度干扰，同等长度的无关技能导致近似损失；另一些模型则因内容误导，即使长度中立，技能内容仍使 Pass@2 下降1.1%至1.4%。进一步分析显示，损失集中于早期简单任务，技能排名跨模型迁移性弱，且反模式规则比示例型内容更有效。这些发现表明，一个有效的技能应被视为特定技能-项目-模型三元组的假设，而非通用资产，将注入重定义为每次部署的路由决策，并要求长度匹配对照与每模型审计成为代理技能评估的最低标准。
原文摘要 · Abstract (English)
Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, and reusable tools. However, because each injected Skill expands the prompt of every query, an effective Skill benchmark must determine not only whether an agent can solve a task, but whether the Skill should have been injected at all. We introduce WebDev-Skills-Bench and use it for a controlled empirical study of 31 public WebDev Skills on 50 Web-Bench projects and 1,000 ordered tasks. The benchmark compares four matched conditions, including a length-matched irrelevant control and leave-one-out component ablations. To isolate Skill effects from prompt-length artifacts, we place only SKILL.md in the prompt while mounting auxiliary files into the agent workspace. Across four models, target Skill injection reduces mean Pass@2 by 1.3% to 4.2%, lowers task completion depth, and increases token cost by 72% to 394%, with gains in only 17% to 36% of Skill-project pairs. Length-matched controls reveal two failure modes: some models are length-distracted, where an equally long irrelevant Skill reproduces most of the loss, while others are content-misled, where prompt length is neutral but Skill content still lowers Pass@2 by 1.1% to 1.4%. Further analysis shows that losses concentrate on easy early tasks, Skill rankings transfer weakly across models, and anti-pattern rules outperform example-heavy content within helpful Skills. These findings recast a matched Skill as a hypothesis about a particular Skill-project-model triple rather than a portable asset, reframing injection as a per-deployment routing decision and making length-matched controls and per-model audits a minimum standard for Agent-Skill evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。