通过添加坐标辅助线索，让视觉语言模型更准地定位界面元素。
How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
- 用坐标轴、网格等辅助线索引导模型推理空间位置。
- 在ScreenSpot-v2上将模型准确率从11.72%提升至95.20%。
- 无需微调，适合快速部署到各类界面理解任务中。
图形用户界面（GUI）定位是构建GUI智能体的基础任务。然而，通用视觉语言模型（VLMs）因缺乏特定优化而表现不佳。本文发现：尽管VLMs在指点游戏（Pointing Game）中表现出显著的隐式空间理解潜力，但在输出显式坐标的任务中却表现欠佳。为弥合这一差距并避免现有微调方法高昂的数据与标注成本，我们提出三种零样本辅助推理方法。通过在输入图像中加入坐标轴、网格和标记交点等显式空间线索，这些方法使VLMs能更好地表达其内在的空间理解能力。我们在四个GUI定位基准上评估了七种开源与专有VLMs的表现。实验结果表明，辅助推理带来显著提升：Mark-Grid Scaffold将Gemini-3.1-Pro在ScreenSpot-v2上的准确率从11.72%提升至95.20%，在ScreenSpot上达到当前最优，在ScreenSpot-v2与UI-I2E-Bench上接近最强微调方法。代码已公开于https://github.com/liweim/AuxiliaryReasoning。
原文摘要 · Abstract (English)
Graphical user interface (GUI) grounding is a fundamental task for building GUI agents. However, general vision-language models (VLMs) struggle with this task due to a lack of specific optimization. We identify a key gap in this paper: while VLMs exhibit significant latent grounding potential, as demonstrated by their performance measured by Pointing Game, they underperform when tasked with outputting explicit coordinates. To address this discrepancy and bypass the high data and annotation costs of current fine-tuning approaches, we propose three zero-shot auxiliary reasoning methods. By providing explicit spatial cues such as axes, grids and labeled intersections as part of the input image, these methods enable VLMs to better articulate their implicit spatial understanding capabilities. We evaluate these methods on four GUI grounding benchmarks across seven open-source and proprietary VLMs. Experimental results show substantial gains from auxiliary reasoning. Mark-Grid Scaffold boosts Gemini-3.1-Pro from 11.72\% under direct inference to 95.20\% on ScreenSpot-v2, achieves state-of-the-art performance on ScreenSpot, and approaches the strongest fine-tuned methods on ScreenSpot-v2 and UI-I2E-Bench. Our code is available at https://github.com/liweim/AuxiliaryReasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。