用海量无标注截图训练GUI智能体,大幅降低人工标注依赖。
GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

- 通过两阶段课程学习,从无标注截图中自建视觉定位能力
- 仅用0.1%标注数据即超越先进系统UI-TARS表现
- 适合想低成本构建跨设备通用GUI代理的研究者
数据是现代智能的基础,推动了当前基础模型的发展。研究者自然希望将这一范式拓展至GUI智能体领域,通过类似方式构建强健的GUI智能体。然而,GUI智能体数据无法直接从互联网获取,导致大规模收集成本高昂且困难。因此,现有GUI智能体普遍存在跨设备泛化能力差、对细粒度界面元素视觉定位能力弱的问题。为应对这一数据挑战,我们提出GUICrafter,一种利用海量无标注截图进行弱监督训练的GUI智能体,显著减少对昂贵人工标注的依赖。GUICrafter采用两阶段渐进式课程学习框架:第一阶段,模型通过大规模未标注截图和网页,利用界面交互中固有的丰富上下文信号,在无需人工标注的情况下学习视觉定位;第二阶段,使用少量高质量数据,通过强化学习校准模型。实验表明,GUICrafter在仅使用0.1%标注数据的情况下,性能达到甚至优于先进系统UI-TARS。在相同标注数据量下,其表现超越所有先前方法,如GUI-R1。代码、数据与模型已公开于https://github.com/fansunqi/GUICrafter。
原文摘要 · Abstract (English)
Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain of GUI agents, hoping to build strong GUI agents through a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, current GUI agents suffer from poor cross-device generalization and limited visual grounding ability for fine-grained GUI elements. As an attempt to address data challenge in GUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores a curriculum learning framework for training GUI agents through two progressive stages. First, the model learns visual grounding from large-scale unannotated screenshots and webpages, leveraging the rich contextual signals inherent in GUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model via reinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。