让GUI自动化工具学会自我检查，避免盲目操作导致错误累积。
Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction
- 引入验证机制，先判断前一步操作是否成功再决定下一步。
- 在真实安卓环境中故障识别率提升42%，修复成功率提高38%。
- 适合需要高可靠性的自动化测试或智能助手场景。
基于视觉语言模型的自主GUI代理常假设环境响应是确定性的，生成操作时未验证先前动作是否成功。在存在网络延迟、渲染延迟和系统中断的真实场景中，这一假设会导致未被发现的操作失败、重复无效行为及灾难性错误累积。此外，由于在线交互成本高且离线数据集缺乏实时反馈，学习鲁棒恢复策略极具挑战。我们提出VeriGUI（验证驱动的GUI代理），显式建模动作结果与恢复策略。VeriGUI采用思考-验证-行动-预期（TVAE）框架以检测失败并引导纠正推理，并设计两阶段训练流程：结合鲁棒SFT与合成失败轨迹，以及带非对称验证奖励的GRPO。我们还构建了基于AndroidControl的鲁棒性基准，用于评估故障识别与纠正能力。实验表明，VeriGUI显著减少失败循环，提升恢复成功率，同时保持与标准任务性能相当的表现。
原文摘要 · Abstract (English)
Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。