arXiv:2608.04828cs.CL2026-08被引 1

测试大模型能否在智能体中正确识别并使用技能,发现当前能力仍严重不足。

Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?

论文配图:Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?
图 1 · 摘自论文原文
  • 设计渐进披露评估框架,检验模型能否主动调用并执行技能
  • 最强模型仅达0.613的综合得分,触发与合规均成瓶颈
  • 技能使用效果依赖智能体架构,非模型固有属性

大型语言模型(LLM)智能体日益依赖技能——结构化文档,规定何时行动、执行何种步骤及允许使用哪些工具。现有评估多关注技能质量或对任务完成的贡献,未检验智能体是否能自主识别并应用相关技能。我们提出Skill-Use基准,通过渐进披露方式评估技能使用:智能体仅获知技能名称与简短描述,需先检索完整规程再执行。Skill-Use区分三个维度:触发(agent是否调用相关技能)、合规(是否忠实遵循规程)、边界(是否避免禁止操作)。综合得分(SU)仅在技能被触发后才计分。该基准包含79个真实技能与177个可执行任务,覆盖九个领域,基于真实文件,运行于隔离Docker沙箱,并采用轨迹评分规则。在两种智能体框架下评估八种模型,结果显示可靠技能使用仍难以实现,最强配置仅得0.613的SU分数。触发与程序合规分别成为独立瓶颈,且得分和模型排名随框架变化,表明技能使用是受框架制约的能力,而非模型固有属性。

原文摘要 · Abstract (English)

Large language model (LLM) agents increasingly rely on skills, structured documents that specify when to act, which procedure to follow, and which tools are allowed. Existing evaluations mostly judge the quality of a skill or its contribution to task success, leaving unexamined whether an agent can recognize a relevant skill and apply it on its own. We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure, where an agent sees only a skill's name and short description and must retrieve the full procedure before following it. Skill-Use separates three facets of skill use. Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations. A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered. Skill-Use pairs 79 real skills with 177 executable tasks across nine domains, each grounded in real files, run in an isolated Docker sandbox, and scored by a trajectory-based rubric. Evaluating eight LLMs under two agent harnesses, we find that reliable skill use remains out of reach, as the strongest configuration reaches an SU of only 0.613. Triggering and procedural compliance fail as independent bottlenecks, and both scores and model rankings shift with the harness, so skill use behaves as a capability conditioned on the harness rather than a fixed property of the model.

智能体技能使用评估基准LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。