Gaia2评测大模型在动态异步环境中的表现,揭示推理与效率的权衡。
Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments
- 设计动态异步环境,模拟真实世界不确定性与时间约束。
- GPT-5达42%通过率但不擅时间敏感任务,开源模型中Kimi-K2最优(21%)。
- 支持动作级验证,适合强化学习训练,可扩展性强。
我们提出Gaia2,一个用于评估大语言模型智能体在真实、异步环境中的基准测试。与以往静态或同步评估不同,Gaia2引入环境独立于智能体行为演化的场景,要求智能体在时间约束下适应噪声和动态事件、解决模糊性并与其他智能体协作。每个场景配备写操作验证器,实现细粒度的动作级评估,可直接用于基于可验证奖励的强化学习。对主流专有及开源模型的评估显示:无模型在所有能力上占优;GPT-5(高)取得最高综合得分42% pass@1,但在时敏任务中表现不佳;Claude-4 Sonnet以牺牲准确性和速度换取成本优势;Kimi-K2在开源模型中领先,达21% pass@1。结果凸显推理、效率、鲁棒性间的根本权衡,并暴露“仿真到现实”差距的挑战。Gaia2基于开源的Agents Research Environments平台构建,易于扩展。我们同步发布基础框架ARE,旨在为社区提供灵活的基础设施,用于开发、评测和训练下一代实用智能体系统。
原文摘要 · Abstract (English)
We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。