arXiv:2606.30084cs.CV2026-06

用单次前向传播实现精准高效的GUI定位,比双次缩放更快更准。

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

论文配图:One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
图 1 · 摘自论文原文
  • 通过跨层证据桥接,在单次前向中保留并重用目标区域信息
  • 在6个基准上达到新最优,最高提升4.1点,延迟降低31.8%
  • 适合追求高精度与低延迟的GUI自动化系统开发者

基于多模态大模型的GUI定位方法通常将目标定位建模为自回归坐标生成,虽能利用MLLM强大的指令跟随与语义理解能力,但需在解码时保持区域级目标证据以满足点击空间精度。诊断分析发现,目标区域意识出现在中间解码层,却未被保留或转化为最终坐标预测。现有ZoomIn类方法通过外部裁剪-重运行的两步流程改善定位,但增加端到端延迟与计算开销。为在不引入额外成本的前提下保留两步缩放的精度优势,我们提出InnerZoom,一种单次前向的跨层证据桥接框架。InnerZoom将原始前向中的目标相关线索转化为紧凑的跨层证据状态,并在后续解码层中持续保留、精炼与重注入,以引导坐标预测。大量实验表明,InnerZoom-4B在全部六个GUI定位基准上取得最优性能:在OSWorld-G上达64.7,在UI-Vision上达40.2,在OSWorld-GR上达73.1,在MMBench-GUI上达87.6,分别超越此前最佳结果4.1、3.2、2.9和2.3点。在受控4B设置下,相比SFT+RL基线平均提升5.3点,优于两步ZoomIn平均1.3点,同时端到端延迟降低31.8%,TFLOPs减少约29%。代码与模型将公开。

原文摘要 · Abstract (English)

MLLM-based GUI grounding methods commonly formulate target localization as autoregressive coordinate generation, enabling models to leverage the strong instruction-following and semantic understanding capabilities of MLLMs. However, this formulation requires the model to retain region-level target evidence while decoding coordinate tokens with the spatial precision demanded by GUI clicking. Our diagnostic analysis reveals that target-region awareness emerges in intermediate decoder layers but is neither retained nor translated into the final coordinate prediction. Existing ZoomIn-style methods address this issue through an external crop-and-rerun pass, which improves localization but increases end-to-end latency and computational cost. To retain the accuracy benefits of two-pass zooming without this extra cost, we propose InnerZoom, a single-forward framework for cross-layer evidence bridging. InnerZoom transforms target-related cues from the original forward pass into a compact cross-layer evidence state, then preserves, refines, and reinjects this state throughout later decoding layers to guide coordinate prediction. Extensive experimental results suggest that InnerZoom-4B achieves state-of-the-art performance on all six GUI grounding benchmarks, obtaining 64.7 on OSWorld-G, 40.2 on UI-Vision, 73.1 on OSWorld-GR, and 87.6 on MMBench-GUI, surpassing the previous best results by 4.1, 3.2, 2.9, and 2.3 points, respectively. Under a controlled 4B setting, InnerZoom improves the same SFT+RL baseline by 5.3 points on average and outperforms two-pass ZoomIn by 1.3 points on average, while reducing end-to-end latency by up to 31.8% and TFLOPs by about 29%. Code and models will be publicly available.

GUI定位多模态模型单次前向跨层融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。