零成本自动化训练图形界面智能体，无需人工标注。
ZeroGUI: Automating Online GUI Learning at Zero Human Cost
- 利用视觉语言模型自动生成任务与奖励，实现无监督学习。
- 在两个平台上的实验显示性能显著提升，超越现有方法。
- 适合想低成本构建自治界面助手的研究者和开发者。
大视觉语言模型的快速发展推动了纯视觉图形界面（GUI）智能体的发展，使其能够感知并操作图形用户界面以自主完成用户指令。然而，现有方法多采用离线学习框架，存在两大核心局限：(1) 严重依赖高质量的人工标注进行元素定位和动作监督；(2) 对动态交互环境适应性差。为此，我们提出 ZeroGUI——一种可扩展的、零人工成本的在线学习框架，用于自动化 GUI 智能体训练。具体而言，ZeroGUI 集成三部分：(i) 基于视觉语言模型的自动任务生成，从当前环境状态生成多样化训练目标；(ii) 基于视觉语言模型的自动奖励估计，无需手工设计评估函数即可判断任务是否成功；(iii) 两阶段在线强化学习，持续与 GUI 环境交互并学习。在 OSWorld 与 AndroidLab 环境中对两个先进 GUI 智能体（UI-TARS 与 Aguvis）的实验表明，ZeroGUI 显著提升了性能。代码已开源：https://github.com/OpenGVLab/ZeroGUI。
原文摘要 · Abstract (English)
The rapid advancement of large Vision-Language Models (VLMs) has propelled the development of pure-vision-based GUI Agents, capable of perceiving and operating Graphical User Interfaces (GUI) to autonomously fulfill user instructions. However, existing approaches usually adopt an offline learning framework, which faces two core limitations: (1) heavy reliance on high-quality manual annotations for element grounding and action supervision, and (2) limited adaptability to dynamic and interactive environments. To address these limitations, we propose ZeroGUI, a scalable, online learning framework for automating GUI Agent training at Zero human cost. Specifically, ZeroGUI integrates (i) VLM-based automatic task generation to produce diverse training goals from the current environment state, (ii) VLM-based automatic reward estimation to assess task success without hand-crafted evaluation functions, and (iii) two-stage online reinforcement learning to continuously interact with and learn from GUI environments. Experiments on two advanced GUI Agents (UI-TARS and Aguvis) demonstrate that ZeroGUI significantly boosts performance across OSWorld and AndroidLab environments. The code is available at https://github.com/OpenGVLab/ZeroGUI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。