让AI更准定位界面元素,关键在显式标注坐标位置。
Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
- 用RULER标记显式坐标,替代模型自行推算位置。
- 在高分辨率界面下准确率提升显著,最高达18.7%。
- 适合需要跨设备可靠操作的自动化系统开发。
GUI定位任务(将自然语言指令映射到像素坐标)对自主代理至关重要,但当前视觉语言模型仍面临挑战。核心瓶颈在于无法可靠地将图像块映射到像素,尤其在训练中未见的高分辨率显示上表现不佳。现有方法直接从视觉特征生成坐标文本标记,迫使模型隐式学习复杂的定位关系,导致准确率下降和错误频发。本文提出两项互补创新:首先引入RULER标记作为显式坐标参考,类似地图上的网格线,使模型可调整而非从零生成坐标;其次采用交错式多尺度旋转位置编码(I-MRoPE),均衡表示宽高维度,缓解标准位置编码的不对称问题。在ScreenSpot、ScreenSpot-V2和ScreenSpot-Pro数据集上的实验表明,该方法在各类分辨率下均取得一致性能提升,尤其在高分辨率界面下效果最显著。通过提供显式空间引导,而非依赖隐式学习,本方法显著增强了跨设备、跨平台的GUI自动化可靠性。
原文摘要 · Abstract (English)
GUI grounding, the task of mapping natural-language instructions to pixel coordinates, is crucial for autonomous agents, yet remains difficult for current VLMs. The core bottleneck is reliable patch-to-pixel mapping, which breaks when extrapolating to high-resolution displays unseen during training. Current approaches generate coordinates as text tokens directly from visual features, forcing the model to infer complex position-to-pixel mappings implicitly; as a result, accuracy degrades and failures proliferate on new resolutions. We address this with two complementary innovations. First, RULER tokens serve as explicit coordinate markers, letting the model reference positions similar to gridlines on a map and adjust rather than generate coordinates from scratch. Second, Interleaved MRoPE (I-MRoPE) improves spatial encoding by ensuring that width and height dimensions are represented equally, addressing the asymmetry of standard positional schemes. Experiments on ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent gains in grounding accuracy, with the largest improvements on high-resolution interfaces. By providing explicit spatial guidance rather than relying on implicit learning, our approach enables more reliable GUI automation across diverse resolutions and platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。