arXiv:2604.21375cs.CLcs.AI2026-04被引 9

解决自动化界面任务中过早结束和死循环问题,提升智能体可靠性。

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

论文配图:VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
图 1 · 摘自论文原文
  • 三模块协同:验证成功、打破循环、按需搜索,动态决策何时停、改、查。
  • 在两个基准上分别达77.5%和61.0%准确率,三模型单次通过超人类水平。
  • 适用于高可靠性要求的自动化测试与运维场景,尤其适合复杂交互任务。

自主图形界面(GUI)智能体面临两大核心挑战:过早终止(未验证即宣告成功)与重复循环(反复执行失败动作)。本文提出VLAA-GUI,一种模块化框架,包含三个核心组件:第一,强制完整性验证器,在每一步结束时检查可观察的视觉证据,通过规则交叉验证拒绝无直接视觉支持的成功声明;第二,强制循环打断器,采用多层级过滤机制:在多次失败后切换操作模式,持续屏幕状态重复时强制策略变更,并将反思信号绑定至策略调整;第三,按需搜索代理,通过具备检索能力的大语言模型查询未知流程,返回纯文本结果。此外,还集成编码代理与定位代理,按需调用以处理代码密集型操作和精准动作定位。在包括Opus 4.5、4.6和Gemini 3.1 Pro在内的五个主流模型上评估,覆盖Linux与Windows任务的两个基准,均取得领先性能(OSWorld:77.5%,WindowsAgentArena:61.0%)。值得注意的是,其中三个模型在单次通过下超越人类表现(72.4%)。消融实验表明,所有组件均能稳定提升强基线表现,弱基线在步数充足时受益更显著;进一步分析显示,循环打断器几乎减少一半的无效步骤。

原文摘要 · Abstract (English)

Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same failing actions without recovery. We present VLAA-GUI, a modular GUI agentic framework built around three integrated components that guide the system on when to Stop, Recover, and Search. First, a mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step -- with an agent-level verifier that cross-examines completion claims with decision rules, rejecting those lacking direct visual evidence. Second, a mandatory Loop Breaker provides multi-tier filtering: switching interaction mode after repeated failures, forcing strategy changes after persistent screen-state recurrence, and binding reflection signals to strategy shifts. Third, an on-demand Search Agent searches online for unfamiliar workflows by directly querying a capable LLM with search ability, returning results as plain text. We additionally integrate a Coding Agent for code-intensive actions and a Grounding Agent for precise action grounding, both invoked on demand when required. We evaluate VLAA-GUI across five top-tier backbones, including Opus 4.5, 4.6 and Gemini 3.1 Pro, on two benchmarks with Linux and Windows tasks, achieving top performance on both (77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance (72.4%) on OSWorld in a single pass. Ablation studies show that all three proposed components consistently improve a strong backbone, while a weaker backbone benefits more from these tools when the step budget is sufficient. Further analysis also shows that the Loop Breaker nearly halves wasted steps for loop-prone models.

GUI自动化智能体框架可靠性优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。