用数据飞轮训练可纠错的GUI智能体,提升操作可靠性。
GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models
- 通过正负样本训练直观评估动作正确性的批评模型。
- 循环收集新数据,使批评模型准确率逐步提升至87.3%。
- 适合需要高可靠性的自动化测试与智能助手场景。
尽管大型视觉语言模型(LVLMs)显著提升了GUI智能体在解析文本指令、理解屏幕内容和执行任务方面的能力,但一个关键挑战依然存在:智能体操作的不可逆性——单一错误操作可能引发灾难性偏差。为解决此问题,我们提出图形用户界面动作批评的数据飞轮系统(GAIA),一种使模型具备迭代式批评能力的训练框架,用于提升基础GUI智能体的测试时缩放(TTS)性能。具体而言,首先使用基础智能体产生的正负动作样本训练一个直观批评模型(ICM),该模型评估智能体意图动作的即时正确性,从而选择成功率更高的操作。随后,初始批评模型引导智能体行动,收集更精细的正负样本,启动自我改进循环。增强后的数据用于训练第二轮具备更强辨别能力的批评模型。我们在多个数据集上进行实验,结果表明所提出的ICM能有效提升各类闭源与开源模型的测试时性能,且随着数据循环利用,性能持续优化。代码、数据集及配套数据表将公开发布于https://github.com/SeerRay-Lab/GAIA。
原文摘要 · Abstract (English)
While Large Vision-Language Models (LVLMs) have significantly advanced GUI agents' capabilities in parsing textual instructions, interpreting screen content, and executing tasks, a critical challenge persists: the irreversibility of agent operations-where a single erroneous action can trigger catastrophic deviations. To address this, we propose the \textbf{G}UI \textbf{A}ction Cr\textbf{i}tic's Dat\textbf{a} Flywheel System (GAIA), a training framework that enables the models to have iterative critic capabilities, which are used to improve the Test-Time Scaling (TTS) of basic GUI agents' performance. Specifically, we train an \textbf{Intuitive Critic Model} (ICM) using positive and negative action examples from a base agent first. This critic evaluates the immediate correctness of the agent's intended actions, thereby selecting operations with higher success probability. Then, the initial critic guides agent actions to collect refined positive/negative samples, initiating the self-improving cycle. The augmented data then trains a second-round critic with enhanced discernment capability. We conduct experiments on various datasets and demonstrate that the proposed ICM can improve the test-time performance of various closed-source and open-source models, and the performance can be gradually improved as the data is recycled. The code, dataset, and accompanying datasheet will be publicly released at https://github.com/SeerRay-Lab/GAIA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。