arXiv:2602.22953cs.AI2026-02被引 8

首次系统评估通用智能体在多场景下的表现,发现架构选择影响大但模型更关键。

General Agent Evaluation

  • 统一协议与评测框架,实现跨环境、跨任务的公平比较
  • 5类架构×5模型×6基准测试,发现顶尖通用智能体媲美定制化方案
  • 开源模型存在性能坍塌问题,不同架构有独特错误模式

通用智能体可在陌生环境中执行任务而无需领域定制。然而,尚无研究系统衡量智能体架构如何影响其在异构协议和多样化未知环境中的表现。本文首次开展系统性研究,对比工具调用、MCP、代码生成和CLI智能体在相同基准与模型上的表现。此前两大障碍限制此类研究:现有评测工具需针对每项基准单独配置或固定协议类型(如BrowserGym的网页协议、Harbor的CLI协议),且基准本身依赖人工编写提示、上下文和集成代码。为此,本文贡献:(1) 统一协议,连接现有基准与智能体协议;(2) 评测框架,支持任意基准与通用智能体及主干模型的对接;(3) 首个开放通用智能体排行榜,覆盖5种智能体架构×5个主干LLM(3个闭源,2个开源)×6个基准,涵盖软件工程、客户服务、深度研究和个人助手。结果表明:(i) 通用智能体无需领域定制即可适应所有测试领域;(ii) 架构选择在单个模型下可导致性能波动达12个百分点,但主干模型选择主导整体表现;(iii) 在6个基准中的4个上,顶尖通用智能体与领先定制化专用智能体无显著差异;(iv) 开源模型表现出“泛化黑洞”现象,在特定架构或基准上持续性能崩溃;(v) 行为失败分析揭示架构特有的错误模式,传统聚合评分无法区分。代码、框架、排行榜与运行记录详见https://www.exgentic.ai。

原文摘要 · Abstract (English)

General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes performance across heterogeneous protocols and diverse unfamiliar environments. This is the first systematic study, comparing tool-calling, MCP, code-generation, and CLI agents on the same benchmarks with the same models. Two gaps blocked such a study: existing harnesses require per-benchmark wiring or fixed protocol classes (web for BrowserGym, CLI for Harbor), and benchmarks themselves expect human-authored prompts, context, and integration glue. To enable this study, we contribute (1) a unifying protocol that bridges existing benchmark and agent protocols; (2) an evaluation harness that surfaces any benchmark to any general-purpose agent and backbone model; and (3) the first Open General Agent Leaderboard of agent configurations, a full factorial over 5 agent architectures x 5 backbone LLMs (three closed-source, two open-weight) x 6 benchmarks spanning software engineering, customer service, deep research, and personal assistance. We find that (i) general agents adapt to every tested domain without per-domain customization; (ii) agent architecture choice swings results by up to 12pp within a single model, yet backbone model choice dominates overall performance; (iii) on 4 of 6 tested benchmarks, top general agents are indistinguishable from the leading heavily-customized domain-specific agents; (iv) open-weight models tested exhibit "generality sinks" absent from frontier closed-source models: they consistently collapse on specific agent architectures or benchmarks; (v) a behavioral failure analysis reveals architecture-distinctive error signatures that aggregate scoring cannot discriminate. Code, harness, leaderboard, and traces are at https://www.exgentic.ai.

通用智能体评测基准大模型评估架构对比

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。