arXiv:2608.24015cs.AI2026-08

提出分步反思机制,让桌面GUI智能体更准判断操作结果。

Reflection with Action-Induced Visual Differences for Desktop GUI Agents

论文配图:Reflection with Action-Induced Visual Differences for Desktop GUI Agents
图 1 · 摘自论文原文
  • 先提取操作引起的视觉变化区域,再验证结果,分离判断逻辑。
  • 在两个基准上分别提升反思准确率7.11%和任务成功率5.94%、4.95%。
  • 适合需要高可靠性桌面操作的自动化系统开发者参考。

规划-执行-反思(POR)框架广泛用于GUI智能体以保持复杂任务中的目标对齐。然而,桌面GUI界面密集且状态变化细微分散,使反思模块承担主要责任:需比较操作前后的屏幕,而规划与执行模块仅基于单一状态推理。现有反思模块将变化检测与结果验证合并为一步,导致证据隐含,决策缺乏依据。为此,本文提出先验证据反思(EFR),一种两阶段反思机制,显式分离操作引发的视觉差异提取与结果验证。EFR通过标记集定位操作位置及候选变化区域,描述并筛选相关变化,最终基于清理后的证据做出判断。该设计使反思更依托屏幕转换,同时降低视觉搜索复杂度与推理负担。在OSWorld-Verified与WindowsAgentArena上的实验表明,EFR将反思准确率提升7.11%,在两个基准上分别带来平均端到端任务成功率5.94%和4.95%的提升。

原文摘要 · Abstract (English)

The Planner-Operator-Reflector (POR) framework is widely used in GUI agents to maintain objective alignment in complex tasks through modular collaboration. However, desktop GUIs introduce a key challenge: large, dense interfaces often exhibit subtle or scattered state changes, placing most of the burden on the reflector, which must compare pre- and post-action screens, while the planner and operator reason over a single state. Existing reflectors collapse change detection and outcome verification into one step, leaving evidence implicit and yielding weakly grounded decisions. To address this limitation, we propose Evidence-First Reflection (EFR), a two-stage reflector that explicitly decouples action-induced visual differences extraction from outcome verification. EFR identifies the action location and candidate changed regions with Set-of-Marks annotations, describes and filters action-relevant changes, and makes the final judgment from the cleaned evidence. This evidence-reasoning decoupled design makes reflection better grounded in screen transitions, while reducing both visual search complexity and reasoning burden. Experiments on OSWorld-Verified and WindowsAgentArena demonstrate that EFR improves reflector accuracy by 7.11%, yielding average end-to-end task success gains of 5.94% and 4.95% on the two benchmarks, respectively.

GUI智能体反思机制桌面自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。