arXiv:2606.29537cs.AI2026-06被引 14

新基准测试挑战长时序真实电脑操作,暴露当前智能体重大短板。

OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

论文配图:OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks
图 1 · 摘自论文原文
  • 构建108个真实世界长流程任务,平均耗时1.6小时、需318步工具调用。
  • 顶尖模型仅完成20.6%任务,暴露出对隐含状态与中途信息的处理缺陷。
  • 适合评估智能体在复杂交互、跨源推理等真实场景下的可靠性。

现有计算机使用基准无法反映真实世界的现实性、复杂性和长时程需求,限制了对前沿智能体局限性的揭示。我们提出OSWorld 2.0,包含108个涵盖日常与专业任务的长时程计算机操作工作流,旨在捕捉复杂且具挑战性的现实现象。每个任务代表一个真实端到端流程,人类平均耗时约1.6小时完成,使用Claude Opus 4.7并启用最大思维模式时平均需318次工具调用,远超OSWorld 1.0的约30次。该基准聚焦真实工作流中常见但先前基准忽视的挑战,包括流式交互、动态环境等交互设计难题,以及跨源推理、隐含状态推断、视觉空间精度等智能体行为挑战。任务基于真实输入文件,并与实际用户状态数据交叉验证,包含独立的安全报告以审计敏感操作。在500步内的二元完成度指标下,启用最大思维和批量工具调用的Claude Opus 4.8表现最佳,但仅完成20.6%任务,部分得分达54.8%;GPT-5.5虽更节省令牌,却在13%处趋于饱和。结果表明,当前智能体仍远未达到专业级计算机操作水平——其问题不在于基础GUI控制或编程,而在于忽略约束、错过中途信息、猜测而非询问用户、跳过验证,在依赖需恢复的隐藏状态的任务上尤其困难。

原文摘要 · Abstract (English)

Existing computer-use benchmarks fail to capture the realism, complexity, and long-horizon demands of real-world computer use, limiting their ability to reveal the limitations of frontier agents. We introduce OSWorld 2.0, a benchmark of 108 long-horizon computer-use workflows across everyday and professional tasks, designed to capture complex and challenging real-world phenomena. Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete and requires an average of 318 tool calls with Claude Opus 4.7 using maximum thinking, compared with about 30 in OSWorld 1.0. OSWorld 2.0 targets challenge phenomena that are common in real workflows yet underrepresented in prior benchmarks, spanning interaction-design challenges such as streaming interaction and dynamic environments, as well as agent-pattern challenges such as cross-source reasoning, implicit-state inference, and visual-spatial precision. Tasks are grounded in authentic input artifacts and cross-referenced against realistic stateful user profile data, and include separate safety reports auditing safety-sensitive execution. Under our primary binary-completion metric at 500 steps, Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score; GPT-5.5 is far more token-efficient yet plateaus near 13%. These results show that current agents are still far from professional-level computer use: rather than stumbling on basic GUI control or coding, they lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification, struggling most when a task hinges on hidden state they must recover.

智能体评测长时程任务真实场景基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。