用自监督强化学习提升GUI智能体，无需人工标注即可自动优化
GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
- 通过预测界面状态变化的初始操作来学习界面动态
- 在四个基准上提升11.2%的自动化准确率，通用性强
- 适合做自动化工具、智能助手的开发者参考
训练用于GUI智能体的视觉语言模型通常依赖大规模标注数据集，其收集成本高且易出错。我们提出K步GUI转换，一种自监督逆动力学任务，使VLM通过预测导致两个GUI状态间转移的初始动作来学习界面动态，无需自然语言指令，可从现有轨迹或自动探索中构建可扩展数据集。基于此任务，我们提出GUI-Shift强化学习框架，结合规则优化与数据过滤以提升VLM性能。在多个VLM骨干网络上，于四个基准（AndroidControl、GUI Odyssey、ScreenSpot-v2、ScreenSpot-Pro）上进行广泛实验，结果表明，使用GUI-Shift训练的模型在GUI自动化和界面定位任务上均表现良好，自动化准确率最高提升11.2%。本研究证明了自监督强化学习可有效利用未标注的GUI轨迹，为替代标注样本提供了可扩展方案。
原文摘要 · Abstract (English)
Training effective Vision-Language Models (VLMs) for GUI agents typically depends on large-scale annotated datasets, whose collection is both labor-intensive and error-prone. We introduce K-step GUI Transition, a self-supervised inverse dynamics task in which VLMs learn GUI dynamics by predicting the initial action that causes a transition between two GUI states. This approach eliminates the need for natural language instructions and enables scalable dataset construction from existing GUI trajectories or automated exploration. Building on this task, we propose GUI-Shift, a reinforcement learning (RL) framework that combines rule-based optimization with data filtering to improve VLM performance. We conduct extensive experiments using multiple VLM backbones across four benchmarks, spanning GUI task automation (AndroidControl, GUI Odyssey) and GUI grounding (ScreenSpot-v2, ScreenSpot-Pro). Our results show that training on GUI-Shift generalizes well to both GUI automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy. This study underscores the potential of self-supervised RL to leverage unlabeled GUI trajectories and offers a scalable alternative to training with annotated samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。