arXiv:2606.20659cs.AIcs.LG2026-06被引 3

提出技能覆盖率,量化大模型智能体使用技能的充分性。

Skill Coverage: A Test Adequacy Metric for Agent Skills

论文配图:Skill Coverage: A Test Adequacy Metric for Agent Skills
图 1 · 摘自论文原文
  • 将技能指令转为可验证的行为约束,基于轨迹判断是否覆盖
  • 实测仅38.66%~45.51%的约束被覆盖,暴露测试盲区
  • 失败约束可精准定位问题,提升技能有效性的改进效率

代理技能编码了大型语言模型代理的可复用过程知识,现有基准表明此类技能能提升任务表现。然而,任务结果无法揭示哪些技能部分被调用,也无法判断代理在调用时是否遵循相关技能指令。这一空白使得难以判断技能是否充分测试,或任务失败是否提供可操作的改进证据。为此,我们提出技能覆盖率,一种基于轨迹的可复用代理技能测试充分性度量。该框架从每项技能提取行为约束,将自然语言指令转化为半结构化约束,规定特定条件下应有代理行为。随后判断每条约束是否被代理轨迹覆盖,并根据行为对已覆盖约束判定通过或失败。在SkillsBench上的应用显示,基准排行榜上的代理轨迹平均仅覆盖38.66%至45.51%的提取约束。我们仅通过强调代理未遵守的原始指令来强化对应技能,再运行相同任务。该强化使五类代理模型的失败任务平均恢复率提升16.0%。结果表明,技能覆盖率既是测试充分性指标,也是观察技能使用行为的细粒度信号。失败任务中的失败约束可提供改进技能有效性的可操作证据。项目网站随论文发布。

原文摘要 · Abstract (English)

Agent skills encode reusable procedural knowledge for large language model (LLM) agents, and existing benchmarks show that such skills can improve task-level performance. However, a task outcome does not reveal which parts of a reusable skill were exercised, nor whether the agent followed the relevant skill instructions when those parts were exercised. This gap makes it unclear whether a skill has been adequately tested, or whether observed task failures provide actionable evidence for improving agent skill effectiveness. To fill this gap, we introduce skill coverage, a trajectory-based test-adequacy metric for reusable agent skills. Our framework extracts skill behavior constraints from each skill, translating natural-language skill instructions into semi-structured constraints that specify the expected agent behavior under particular conditions. It then determines whether each constraint is covered by an agent trajectory and, for covered constraints, assigns a Pass or Fail verdict according to the agent behavior. We apply this framework to SkillsBench. The results show that agent trajectories on the benchmark leaderboard cover only 38.66 to 45.51% of the extracted skill behavior constraints on average. We then use Fail verdicts to strengthen the corresponding skill content only by emphasizing the original instructions that the agent failed to follow, and run the same tasks with the strengthened skills. This emphasis yields an average 16.0% recovery rate of the failed tasks across the five agent-model rows. These results show that skill coverage is both a test-adequacy metric and a fine-grained signal for observing skill-use behavior. In failed tasks, failed constraint labels provide actionable evidence for improving agent skill effectiveness. A project website accompanies the paper.

智能体测试评估技能覆盖率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。