用可执行测试评估企业技能的实际价值，而非仅看文档描述。
Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
- 通过真实运行对比有无技能的代理表现，量化技能实际贡献。
- 平均综合增益达0.2134，72.8%的案例中技能带来正向提升。
- 适合需要验证技能落地效果的企业AI团队使用。
企业代理项目正从原型走向生产，可复用的技能、工具和工作流包需以证据而非文字描述来评估。现有评审机制多关注结构、风格与安全，却无法回答部署问题：同一模型、沙箱和评分策略下，该能力包是否真正帮助代理完成任务？我们提出ACES（Agentic Continuous Evaluation of Skills），一个原生嵌入代码库的框架，将技能作为可执行代理资产进行评估。它通过配对实验，在有无目标技能条件下运行真实任务，将轨迹标准化为Agent Trajectory Interchange Format（ATIF），评估六项默认运行指标，并报告“技能增益”（Skill Lift）——即在固定任务、资源环境与评分器下，该技能带来的实际价值。相同协议支持产品级任务套件，用于对比基线、单技能、技能包、团队技能及插件。基于内部企业仓库和公开目录中的145个真实技能，仅扫描的门禁虽能发现作者问题，但与人工评分的相关性极低（斯皮尔曼ρ=0.14）。在64个生产技能中的58个、4个主要测试平台共947个配对案例中，平均综合技能增益为0.2134（95%置信区间[0.1967, 0.2301]），仅结果指标增益均值为0.1799。最大改进体现在技能执行、行为检查和效率上，这些是扫描无法观测的流程信号，揭示了发现、路由、流程遵循与工具使用情况。方法开源实现已发布于NVIDIA SkillEvaluator。
原文摘要 · Abstract (English)
Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman $ρ= 0.14$). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95\% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8\% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。