arXiv:2608.01004cs.LGcs.AI2026-08中稿 · the SERI 2026 Indu…

用能力分类法自动筛选回归测试集,提升平台评估效率。

Who Belongs in the Eval Set? A Capability-Taxonomy-Driven Pipeline for Curating Regression Eval Sets in Agent-Extensibility Platforms

论文配图:Who Belongs in the Eval Set? A Capability-Taxonomy-Driven Pipeline for Curating Regression Eval Sets in Agent-Extensibility Platforms
图 1 · 摘自论文原文
  • 基于能力分类体系,结合规则与大模型判断每条查询是否该保留
  • 通过查询覆盖度和执行质量评分,动态优化测试集内容
  • 适合需要高效管理海量客户测试数据的AI平台团队

在支持代理扩展的平台中,每个客户带来的评估集都需适配其领域,但平台的回归测试集受查询数量上限制约。现有框架多为客户端视角,且不处理持续流入的评估集流。本文提出一种基于能力分类的测试集筛选流水线,应用于Microsoft 365 Copilot中的声明式代理。输入包括代理规格和客户评估集,将每条查询映射到平台拥有的能力分类体系,输出“保留、丢弃、替换或人工审核”的决策。核心理念是:健康的回归集应以最小规模覆盖最广的能力组合。系统包含三部分:1)混合使用规则提取与大模型语义推理,生成每条查询对各项能力的判定;2)调用质量(IQ)评分器衡量查询对各能力的触发深度;3)合并器通过规则级联比较新查询与现有测试集的覆盖度与质量,由保守的策展人仅建议剔除项。该机制不依赖特定分类体系,适用于任意具有类型化能力分类的问题,且可随自身输出结果动态演进。

原文摘要 · Abstract (English)

Platform teams hosting agent-extensibility surfaces face a regression-economics paradox: every onboarding customer ships an evaluation set tuned to their domain, but the platform's regression set must live under a hard query-count ceiling bounded by release cadence. To our knowledge, no published industrial pipeline addresses this platform-side curation problem: existing evaluation frameworks are customer-side, and benchmark-compression work treats benchmarks as fixed pools rather than streams of incoming sets. We describe a capability-taxonomy-driven curation pipeline applied to declarative agents with custom actions in Microsoft 365 Copilot. It takes an agent specification and a customer's eval set as input, projects each query into a platform-owned capability taxonomy, and outputs per-query decisions (admit, drop, swap, or human review), under the philosophy that a healthy regression set is the minimal set of queries capturing the maximal spread of capability signatures -- distinct combinations of capabilities a query exercises together. Three components instantiate this: a classifier producing per-(query, capability) verdicts via a hybrid of deterministic specification-based extraction and large-language-model (LLM) semantic inference; an Invocation Quality (IQ) rater scoring how thoroughly a query exercises each capability, so a new query sharing a signature with an existing entry can still be recognized as a better test and displace it; and a consolidator comparing incoming queries against the regression set on coverage and quality through a rule-based decision cascade, backed by a conservative curator that only suggests evictions. The mechanism is taxonomy-agnostic and applies to any regression eval-set curation problem with a typed capability taxonomy, including taxonomies that evolve in response to the very evidence the pipeline surfaces.

评估集构建能力分类AI平台

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。