从零开始训练的界面定位模型,精准操作能力达新高。
POINTS-GUI-G: GUI-Grounding Journey
- 从弱感知基础模型出发,通过数据工程与训练优化实现端到端界面定位
- 在多个基准上刷新纪录:最高达95.7分(ScreenSpot-v2)
- 首次证明强化学习可显著提升界面感知精度,适合自动化任务研究者
视觉语言模型的快速发展推动了GUI智能体的兴起,有望自动完成在线购物、航班预订等复杂数字任务,减轻重复性工作负担。作为关键基础能力,GUI定位需精确识别界面元素(如文字、图标)以实现点击、输入等操作。不同于以往基于已有强空间感知能力模型(如Qwen3-VL)的微调方法,本文从仅具基础定位能力的模型(如POINTS-1.5)出发,构建POINTS-GUI-G-8B,实现领先性能:ScreenSpot-Pro达59.9分,OSWorld-G达66.0分,ScreenSpot-v2达95.7分,UI-Vision达49.9分。成功源于三大要素:(1) 数据工程优化,统一多源开源数据格式,并采用增强、筛选与难度分级策略;(2) 训练策略改进,持续微调视觉编码器以提升感知精度,并保持训练与推理分辨率一致;(3) 基于可验证奖励的强化学习,首次证实其在感知密集型界面定位任务中显著提升精度,且任务本身天然支持高准确奖励反馈。
原文摘要 · Abstract (English)
The rapid advancement of vision-language models has catalyzed the emergence of GUI agents, which hold immense potential for automating complex tasks, from online shopping to flight booking, thereby alleviating the burden of repetitive digital workflows. As a foundational capability, GUI grounding is typically established as a prerequisite for end-to-end task execution. It enables models to precisely locate interface elements, such as text and icons, to perform accurate operations like clicking and typing. Unlike prior works that fine-tune models already possessing strong spatial awareness (e.g., Qwen3-VL), we aim to master the full technical pipeline by starting from a base model with minimal grounding ability, such as POINTS-1.5. We introduce POINTS-GUI-G-8B, which achieves state-of-the-art performance with scores of 59.9 on ScreenSpot-Pro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UI-Vision. Our model's success is driven by three key factors: (1) Refined Data Engineering, involving the unification of diverse open-source datasets format alongside sophisticated strategies for augmentation, filtering, and difficulty grading; (2) Improved Training Strategies, including continuous fine-tuning of the vision encoder to enhance perceptual accuracy and maintaining resolution consistency between training and inference; and (3) Reinforcement Learning (RL) with Verifiable Rewards. While RL is traditionally used to bolster reasoning, we demonstrate that it significantly improves precision in the perception-intensive GUI grounding task. Furthermore, GUI grounding provides a natural advantage for RL, as rewards are easily verifiable and highly accurate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。