arXiv:2608.06891cs.AI2026-08

通过可解释的文档分析,精准评估智能体技能质量并指导改进。

SkillEval: Decomposing Agent Skill Quality into Interpretable Signals

论文配图:SkillEval: Decomposing Agent Skill Quality into Interpretable Signals
图 1 · 摘自论文原文
  • 基于模型隐空间学习技能文档各属性的可解释评分方向
  • 能准确区分不同质量技能,且与下游任务表现高度相关
  • 可定位技能缺陷,支持针对性优化,适合技能开发人员

智能体技能是可复用的过程知识,有助于解决特定任务。随着应用扩展,评估技能质量变得愈发重要。现有方法多通过下游任务表现衡量技能质量,但仅反映技能与任务的兼容性,无法揭示具体改进方向。我们发现, exttt{SKILL.md} 文档的通用特性对技能质量有重要影响。为此提出 extbf{SkillEval},一个可解释的文档级技能评估框架。该框架为每个质量属性学习固定、可检查的评分方向,通过投影新技能表示实现评分,并消除长度、格式等无关特征干扰,确保每项得分聚焦于目标语义属性。在受控质量测试中,SkillEval 能可靠区分不同质量技能,且其评分与下游任务成功率高度一致,可提前预判技能有效性。进一步用于诊断技能文档弱点并指导修改,优化后技能在下游任务中的通过率显著提升。

原文摘要 · Abstract (English)

Agent skills provide reusable procedural knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality by testing whether a skill improves performance on specific downstream tasks. However, a reusable skill may apply to multiple task scenarios. Downstream evaluation mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspect of the skill should be improved. We find that general properties of the \texttt{SKILL.md} document play an important role in skill quality. To evaluate these properties, we propose \textbf{SkillEval}, an interpretable framework for document-level skill evaluation. SkillEval evaluates each property using a fixed and inspectable scoring direction, producing interpretable scores. It further measures and reduces the influence of unrelated document features, such as length and formatting, so that each score captures its intended semantic property more specifically. Specifically, SkillEval learns an interpretable direction for each quality property from controlled positive--negative skill pairs in the hidden representation space of the model, and scores a new skill by projecting its representation onto these fixed directions. We use SkillEval to evaluate skills in controlled quality tests and show that SkillEval reliably distinguishes skills of different quality. In addition, SkillEval scores closely reflect downstream task performance, providing an early indication of whether a skill is likely to help an agent complete a task. We further explore SkillEval for diagnosing weaknesses in skill documents and guiding targeted revisions. The revised skills improve the targeted properties and achieve higher pass rates on downstream tasks.

智能体技能可解释评估文档分析质量优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。