让界面智能体学会纠错回溯，提升任务成功率。
BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism
- 引入验证、判断、反思模块，自动检测并回溯错误操作
- 在移动和自动化界面基准上，成功率与步骤准确率均提升
- 专为回溯设计数据集，关注动作执行后的页面状态
图形用户界面（GUI）智能体因其在多步交互中完成任务的能力而受到广泛关注。然而，现有智能体主要聚焦于单个动作的准确性，缺乏有效的错误检测与恢复机制。为此，我们提出 BacktrackAgent，一个具备回溯能力的鲁棒框架。该框架包含验证器、判断器和反思器三个模块，用于错误检测与恢复，并引入判断奖励进一步优化性能。此外，我们构建了一个专为回溯机制设计的训练数据集，考虑动作执行后的结果页面。实验表明，BacktrackAgent 在 Mobile3M 和 Auto-UI 基准上均显著提升了任务成功率与步骤准确率。代码与数据将在录用后公开。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents have gained substantial attention due to their impressive capabilities to complete tasks through multiple interactions within GUI environments. However, existing agents primarily focus on enhancing the accuracy of individual actions and often lack effective mechanisms for detecting and recovering from errors. To address these shortcomings, we propose the BacktrackAgent, a robust framework that incorporates a backtracking mechanism to improve task completion efficiency. BacktrackAgent includes verifier, judger, and reflector components as modules for error detection and recovery, while also applying judgment rewards to further enhance the agent's performance. Additionally, we develop a training dataset specifically designed for the backtracking mechanism, which considers the outcome pages after action executions. Experimental results show that BacktrackAgent has achieved performance improvements in both task success rate and step accuracy on Mobile3M and Auto-UI benchmarks. Our data and code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。