arXiv:2608.03270cs.CVcs.AI2026-08

通过逐步聚焦视觉区域,让AI更准地点击复杂界面中的目标控件。

GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

论文配图:GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
图 1 · 摘自论文原文
  • 先粗后细:用OCR和组件定位生成坐标参考,分步放大查看区域
  • 多轮观察:每轮根据指令和当前视图选择新视角,直到精准定位
  • 通用适配:兼容多种大模型,对高密度界面点击准确率提升24.9%

GUI grounding 将自然语言指令映射为点击位置,是构建可靠 GUI 代理的关键。在高分辨率、密集布局的界面中,视觉-语言模型(VLM)虽能识别目标控件,却常无法精确定位以完成操作。现有方法虽提供局部化辅助,仍依赖直接点击预测,导致视觉模糊或初始估计误差传递至最终结果。本文提出 GUI-Lens,一种基于通用 VLM 的粗到细定位框架,使模型通过主动视觉观察确定目标。具体而言,GUI-Lens 从截图中提取 OCR 文本与检测到的 UI 组件及其位置,作为坐标参考。结合指令、当前视图及这些参考,VLM 选择下一视角的区域与缩放比例,裁剪并放大后提供更精细视觉信息。该过程持续进行,逐步聚焦直至目标明确。每次提出的裁剪区域与点击位置均依据指令验证,最终将局部坐标映射回原始屏幕坐标。在四个 GUI grounding 基准和三种通用 VLM 后端上的实验表明,GUI-Lens 最多提升 24.9 个百分点的整体定位准确率,并在 GPT-5.5 上达到当前最优性能。

原文摘要 · Abstract (English)

GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on four GUI grounding benchmarks and three general-purpose VLM backends show that GUI-Lens improves overall grounding accuracy by up to 24.9 percentage points and achieves state-of-the-art performance with GPT-5.5.

GUI 接地视觉推理多轮观察大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。