arXiv:2605.08261cs.SEcs.AI2026-05

1MB脚本竟胜过前沿模型,暴露了电脑使用评估的系统性漏洞。

Computer Use at the Edge of the Statistical Precipice

论文配图:Computer Use at the Edge of the Statistical Precipice
图 1 · 摘自论文原文
  • 用简单回放脚本测试,发现无需观察界面也能超越先进模型。
  • 在确定性环境下,回放成功率等于原模型的pass@k值。
  • 提出PRISM原则与DigiWorld基准,支持320万种真实配置验证。

在交互式环境中评估计算机使用代理(CUA)存在方法论陷阱,当前尚未系统解决。我们发现,一个仅1MB的回放脚本,在不观察屏幕的情况下,仍能在主流静态基准上超越前沿模型;并证明其预期成功率在确定性环境下恰好等于源代理的pass@k值。我们追溯失败根源为两大原因:非严谨环境设计(静态、未沙盒化或不可靠验证)和非严谨评估方法(盲目聚合与误用pass@k评估状态相关界面操作)。为此,我们提出PRISM五项环境设计原则(特权验证、真实环境、配置完整性校验、沙盒执行、多因子可变性),并在DigiWorld中实现,该基准包含15个真实沙盒化移动端应用,支持超过320万种已验证的独特配置。同时,我们开发了结合威尔逊区间与分层自举法的聚合框架,生成正确反映嵌套结构的置信区间,实证有效。结果表明,严谨环境设计与严格评估方法并非可选优化,而是有意义CUA研究的前提。

原文摘要 · Abstract (English)

Evaluating Computer Use Agents (CUAs) on interactive environments is fraught with methodological pitfalls that the field has yet to systematically address. We show that a 1MB replay script that blindly executes a recorded action sequence without ever observing the screen outperforms frontier models on prominent static benchmarks, and prove that its expected success rate is exactly equal to the source agent's pass@k in deterministic environments. We trace this and other failures to two root causes: non-principled environment design (static, unsandboxed, or unreliably verified environments) and non-principled evaluation methodology (naive aggregation and misuse of pass@k for stateful UI interactions). To address the first, we propose PRISM, five design principles for CUA environments (privileged verification, realistic environments, integrity-checked configurations, sandboxed execution, and multifactorial variability) and instantiate them in DigiWorld, a benchmark of 15 realistic sandboxed mobile applications able to evaluate agents in over 3.2 million verified unique configurations. To address the second, we develop an aggregation framework pairing Wilson score intervals with hierarchical bootstrap, producing confidence intervals that correctly account for the nested structure of CUA benchmarks, as we empirically demonstrate. All together, we show that principled environment design and rigorous evaluation methodology are not optional refinements but prerequisites for meaningful CUA research.

智能体评估基准测试沙盒环境方法论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。