测试大模型智能体在多任务环境下的实时扩展能力，发现现有方法效果有限。
Benchmark Test-Time Scaling of General LLM Agents
- 构建统一评估框架，测试智能体在搜索、编程、推理等多领域表现。
- 十款主流智能体在通用环境下性能显著下降，且两种扩展方式均无效。
- 揭示顺序扩展受限于上下文长度，并行扩展存在验证差距问题。
大语言模型智能体正被期待作为能够处理开放式用户请求的通用系统。现有基准主要聚焦于特定领域的环境以开发专用智能体，而评估通用智能体则需要更贴近现实的场景，要求其在统一环境中具备跨多种技能和工具的操作能力。本文提出 General AgentBench，一个用于在搜索、编程、推理与工具使用等多个领域评估通用大语言模型智能体的统一基准。利用该基准，我们系统研究了在顺序扩展（迭代交互）和并行扩展（采样多条路径）下的测试时缩放行为。对十款领先智能体的评估显示，从领域专用评测转向通用设置后，性能出现显著下降。此外，我们发现两种缩放方法在实践中均未能带来有效提升，原因在于两个根本限制：顺序扩展受制于上下文长度上限，并行扩展存在验证差距。代码已公开于 https://github.com/cxcscmu/General-AgentBench。
原文摘要 · Abstract (English)
LLM agents are increasingly expected to function as general-purpose systems capable of resolving open-ended user requests. While existing benchmarks focus on domain-aware environments for developing specialized agents, evaluating general-purpose agents requires more realistic settings that challenge them to operate across multiple skills and tools within a unified environment. We introduce General AgentBench, a benchmark that provides such a unified framework for evaluating general LLM agents across search, coding, reasoning, and tool-use domains. Using General AgentBench, we systematically study test-time scaling behaviors under sequential scaling (iterative interaction) and parallel scaling (sampling multiple trajectories). Evaluation of ten leading LLM agents reveals a substantial performance degradation when moving from domain-specific evaluations to this general-agent setting. Moreover, we find that neither scaling methodology yields effective performance improvements in practice, due to two fundamental limitations: context ceiling in sequential scaling and verification gap in parallel scaling. Code is publicly available at https://github.com/cxcscmu/General-AgentBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。