arXiv:2605.27898cs.AI2026-05被引 3

统一评估大模型智能体能力,让不同测试结果可比可复现。

UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities

论文配图:UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
图 1 · 摘自论文原文
  • 将评测拆解为指令-工具-环境三元组,用统一流程执行
  • 15个模型在40万次运行中表现差异大,排名常反转
  • 支持离线模式,适合研究者对比模型真实行为

智能体评测正被广泛用于跨领域比较大语言模型,但现有分数反映的是完整模型-工具-环境配置的综合表现,而非模型本身。评测包通常捆绑特定提示、工具协议、调度逻辑和动态外部资源,导致跨评测结果受实现细节与资源条件影响。我们提出UniACE,一种面向模型的统一评估框架,在明确一致的执行条件下进行评测。UniACE将每个评测任务表示为指令-工具-环境三元组,通过共享的、任务无关的执行引擎在隔离的运行环境中执行模型,并保留原始成功标准。对于依赖动态资源的任务,提供可选离线模式,以预收集的快照替代实时访问。其评估协议进一步标准化了效率测量、执行记录和基于轨迹的失败归因。我们迁移了涵盖24个领域的7个评测基准,对15个模型进行了超过40万次滚动测试,消耗50亿token。与原实现相比,得分变化显著且模型排名多次反转;在线与离线运行对比显示,可获取证据及其表达方式对结果有重大影响。在统一的UniACE配置下,效率与失败特征揭示了仅凭任务成功率无法捕捉的任务相关模型行为。这些发现表明应明确报告评测配置,以实现更可解释、可复现的跨评测比较。代码与数据集见:https://github.com/whfeLingYu/A-Unified-Framework-for-the-Evaluation-of-LLM-Agentic-Capabilities, https://huggingface.co/datasets/whfeLingYu/Unified_Agent_Framework。

原文摘要 · Abstract (English)

Agent benchmarks are increasingly used to compare large language models (LLMs) across domains, yet a reported score reflects a complete model--harness--environment configuration rather than the model alone. Benchmark packages couple native tasks with specific prompts, tool protocols, orchestration logic, and sometimes dynamic external resources, making cross-benchmark comparisons sensitive to implementation and resource conditions. We present UniACE, a unified framework for model-centric evaluation under an explicit, common execution condition. UniACE represents each benchmark as an instruction--tool--environment triplet, executes LLMs through a shared, task-agnostic harness in isolated per-task runtimes, and preserves native success criteria. For tasks that rely on dynamic resources, an optional offline mode replaces live access with fixed, pre-collected snapshots. Its evaluation protocol further standardizes efficiency measurement, execution records, and trace-based failure attribution. We migrate 7 benchmarks spanning 24 domains and evaluate 15 models in more than 400K rollouts consuming 5B tokens. Comparisons with source implementations show large bidirectional score changes and model-ranking reversals, while matched online and offline runs reveal substantial sensitivity to accessible evidence and its representation. Under the shared UniACE configuration, efficiency and failure profiles expose task-dependent model behaviors hidden by task-success scores alone. These findings motivate reporting agent benchmark outcomes as properties of an explicit evaluation configuration, enabling more interpretable and reproducible cross-benchmark comparisons. Codes and benchmarks at are available at https://github.com/whfeLingYu/A-Unified-Framework-for-the-Evaluation-of-LLM-Agentic-Capabilities, https://huggingface.co/datasets/whfeLingYu/Unified_Agent_Framework.

智能体评测大模型评估可复现性统一框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。