测试电脑助手是否能理解桌面操作的因果变化，发现模型普遍表现不佳。
Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
- 构建离线阶梯级基准，验证模型对操作前后状态变化的理解能力。
- 最佳模型在无干扰场景下准确率仅65.1%，存在明显认知盲区。
- 适合研究桌面智能体可靠性、错误恢复与任务验证的团队使用。
计算机使用智能体（CUAs）越来越多地通过桌面GUI完成长周期任务。现有基准主要衡量最终任务成功率或单帧定位，均无法分离模型是否能重建操作产生的因果性、任务相关状态转移——这对排除过时观测、验证进展和故障恢复至关重要。由于推理、远程输入、应用渲染和截图捕获异步，下一观测可能延迟、遮挡、瞬时或无关，易被误判为进展并带入后续规划。本文提出Desktop-Delta Bench（DDB），一个包含2,013个经人工验证实例的离线阶梯级基准，来自约15个应用、50个任务领域的新型多应用Linux轨迹。DDB通过两类互补任务覆盖三种失败维度：状态验证、源追踪与上下文感知控制。包括463个三帧时序排序实例（含105个跨轨迹干扰项），以及1,550对前-后状态对，标注5种操作及其内容。评估8个开源与闭源模型家族，在32种排序与16种单动作设置中，观察到持续差距。排序任务尚未饱和：最佳非干扰与干扰项精确匹配率分别为65.1%与65.7%。任务上下文使干扰识别提升6.9个百分点，但非干扰精确匹配下降2.2点；错误分析显示系统性复制给定的A-B-C顺序。单动作结果表明，推断操作类别比定位其位置更难：点击F1的得分0.96，拖拽仅为0.76，且拖拽位置通常定位良好。DDB弥补了端到端基准缺失的诊断层，介于GUI定位与最终任务成功之间，助力桌面CUA的验证、可靠性和恢复能力改进。
原文摘要 · Abstract (English)
Computer-use agents (CUAs) increasingly act through desktop GUIs to complete long-horizon tasks. Current benchmarks primarily measure end-task success or single-frame grounding. Neither isolates whether a model can reconstruct the causal, task-relevant transition produced by an action- crucial for rejecting stale observations, verifying progress, and recovering from failure. This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous: the next observation may be delayed, occluded, transient, or unrelated, then misread as progress and carried into subsequent planning. We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories across ~15 applications and 50 task domains. DDB trajectories targets 3 failure dimensions- state verification, source tracking, and context-aware control- through 2 complementary tasks: 463 3-frame temporal-ordering instances, including 105 with a cross-trajectory decoy, and 1,550 before-after pairs labeled from 5 actions + its payload. We evaluate 8 closed and open-source model families across 32 ordering and 16 single-action settings, observing consistent gaps. Ordering remains unsaturated: best non-decoy and decoy exact-match rates are 65.1% and 65.7%. Task context improves decoy identification by 6.9 percentage points but reduces non-decoy exact match by 2.2 points; error analysis reveals systematic copying of the presented A-B-C order. Single-action results show that inferring the action family is harder than locating it: click F1 is 0.96 vs, 0.76 for drag, while recognized drags are generally localized well. DDB, thus, complements end-to-end benchmarks by filling the missing diagnostic layer between GUI grounding and final task success, enabling targeted improvements to desktop CUA verification, reliability, and recovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。