arXiv:2607.02032cs.AIcs.CL2026-07

用少量测试题预测大模型智能体性能,省钱省时。

PACE: A Proxy for Agentic Capability Evaluation

论文配图:PACE: A Proxy for Agentic Capability Evaluation
图 1 · 摘自论文原文
  • 从基础能力测试中选关键题目构建代理评估集
  • 预测准确率超85%,误差低于4%
  • 适合模型选型与开发中的快速性能评估

在SWE-Bench和GAIA等智能体基准上评估大模型成本高昂,单次评估需数千美元且耗时数日。相比之下,非智能体基准(如推理、代码生成)测试快速廉价。本文提出PACE框架,通过从现有原子能力测试中精选少量实例,构建能可靠预测智能体表现的代理基准。该框架结合目标相关局部选择与全局信息最优选择策略,建立回归模型,将模型在小规模源任务上的得分映射到目标智能体基准得分。应用至4个智能体基准,形成可落地的PACE-Bench。14个模型、4个智能体基准、19个非智能体基准的实验表明,其在留一交叉验证下平均绝对误差低于4%,斯皮尔曼相关系数高于0.80,配对排序准确率约85%,成本不足原评估的1%。进一步分析揭示各智能体基准的独特能力要求。该方法使开发者能在模型训练、选型与路由中低成本获得可靠性能估计。

原文摘要 · Abstract (English)

Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.

智能体评估代理基准大模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。