通过重做预填充阶段,提升视觉语言模型的GUI定位准确率。
What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs

- 引入注意力引导的二次预填充,重构目标候选选择过程。
- 在5个基准上提升4.3%准确率,无需额外训练。
- 适合需要高精度界面元素定位的AI交互研究者。
现有免训练的GUI定位方法通常依赖多次推理,如迭代裁剪或候选聚合,但每次前向传播仍独立解析指令与视觉布局,无法实现视觉标记间的渐进交互。本文研究了视觉语言模型(VLMs)中GUI定位的过程,发现一个被忽视的瓶颈:定位遵循两阶段范式——预填充阶段决定候选UI元素,解码阶段则细化最终坐标。这种不对称性使预填充成为关键步骤,因候选选择错误难以在解码阶段纠正。基于此,我们提出Re-Prefill,一种免训练方法,通过引入注意力引导的第二预填充阶段来重构推理过程。具体而言,提取查询位置(即最终标记)在各层中持续获得高注意力的视觉标记,作为初步目标假设,并将其与指令隐状态一同拼接回输入,使模型在生成坐标前深度重思决策。在四个VLM和五个基准(ScreenSpot-Pro、ScreenSpot-V2、OSWorld-G、UI-Vision、MMBench-GUI)上的实验表明,该方法在不增加训练成本的前提下取得一致提升,最高达4.3%。代码将开源于https://github.com/linjiaping1/Re-Prefill。
原文摘要 · Abstract (English)
Existing training-free approaches for GUI grounding often rely on multiple inference runs, such as iterative cropping or candidate aggregation, to identify target elements. Despite this additional computation, each forward pass still independently interprets the instruction and parses the visual layout, without enabling progressive interaction among visual tokens. In this paper, we study what happens during GUI grounding in Vision-Language Models (VLMs) and identify a previously overlooked bottleneck. We show that grounding follows a two-stage paradigm: the prefill stage determines candidate UI elements, while the decoding stage subsequently refines the final coordinates. This asymmetry establishes prefill as the critical step, as errors in candidate selection cannot be effectively corrected during decoding. Based on this observation, we propose Re-Prefill, a training-free method that revisits inference by introducing an attention-guided second prefill stage to refine target selection. Specifically, visual tokens that consistently receive high attention from the query position, i.e., the final token, across layers are extracted as a preliminary target hypothesis and appended to the input, together with the instruction hidden states, enabling the model to deeply re-think its decision before coordinate generation. Experiments across four VLMs and five benchmarks, including ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G, UI-Vision, and MMBench-GUI, demonstrate consistent improvements without additional training, with gains of up to 4.3% on ScreenSpot-Pro. Code will be available at https://github.com/linjiaping1/Re-Prefill.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。