让智能体技能测试从主观试用变为可复现的自动化流程。
AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows
- 通过开发者定义的测试契约，自动执行并验证技能表现。
- 将虚假的100%通过率转为可审计的首次失败信号。
- 分离执行与评分模块，防止智能体自纠错导致评估失真。
现代智能体系统依赖可安装的技能包（包含自然语言和代码），使大模型能完成特定领域任务。随着技能库增长，每次变更都需要自动化质量评估，但当前评估仍以主观试用为主：开发者让智能体运行技能，观看演示后形成印象。这种做法缺乏可复现性和版本间可比性，且在市场环境中难以扩展——一次回归可能悄然破坏数十个下游流程。本文提出AEVAL（Agentic Evaluation），一个集成于CI的确定性测试框架，替代传统主观评估。每次技能变更触发测试事件：技能在自动化执行器中按开发者声明的评估契约运行，生成结构化、有证据支撑的质量信号，供下游CI处理。核心是执行器与评分器的结构分离，避免智能体在执行中自我修正后自评通过的隐蔽错误。贡献包括：(i) 基于变更触发的确定性评估协议，含每项技能的契约与每轮结果的模式；(ii) 将自纠正偏差形式化为智能体评估中的独立失效模式；(iii) 执行器/评分器分离机制，采用首次尝试评分规则并显式追踪自修正行为；(iv) 分级的、基于证据的修复建议方案（LV1因果，LV2质量），以行内合并请求评论形式呈现。在多个代理SDK的真实生产环境中验证，AEVAL成功将虚假的100%通过率转化为可复现的首次失败信号，并保留每轮修复的可审计记录。
原文摘要 · Abstract (English)
Modern agentic systems increasingly rely on skills: installable packages of natural language and code that teach an LLM agent to perform a domain task. As skill repositories grow, developers need automated quality signals on every change, yet evaluation today is largely anecdotal: a developer asks an agent to "try the skill," watches a demo, and forms a subjective impression. This yields neither reproducibility across runs nor comparability across versions, and scales poorly to marketplaces where one regression can silently break dozens of downstream workflows. We present AEVAL (Agentic Evaluation), a CI-integrated framework that replaces this practice with a deterministic, reproducible test pipeline for agentic skills. Every skill change triggers a test event: the skill runs against a developer declared evaluation contract inside an automated executor, emitting a structured, evidence grounded quality signal that downstream CI can route on. A key ingredient is a structural separation between executor and grader, preventing a subtle but pervasive failure mode: an agent that silently self-corrects during execution and then grades its own patched outputs as passing. Our contributions are: (i) a deterministic, change-triggered evaluation protocol with per-skill contracts and per-run artifact schemas; (ii) a formalization of self correction bias as a distinct failure mode of naive agentic evaluators; (iii) an executor/grader separation with a first-attempt grading rule and explicit self-correction tracking; and (iv) a tiered, grounded evidence fix suggestion scheme (LV1 causal, LV2 quality) posted as inline merge-request comments. Validated on real skills in a production agentic stack across multiple agent SDKs, AEVAL converts spurious 100% pass rates into reproducible first-attempt fail signals with an auditable record of every executor fix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。