arXiv:2605.02630cs.CV2026-05被引 1

让AI看懂高分辨率界面,自动聚焦不确定区域提升定位精度

AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding

论文配图:AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
图 1 · 摘自论文原文
  • 用注意力困惑度衡量视觉不确定性,生成方向性概率场
  • 在ScreenSpot数据集上提升通用与专用模型定位准确率
  • 无需训练,适合高分辨率界面的智能搜索任务

视觉语言模型(VLMs)已实现将自然语言指令转化为可执行屏幕坐标的自主GUI代理。然而,在高分辨率界面中,密集布局和微小交互元素导致现代显示分辨率与模型输入限制之间存在分辨率差距,使定位性能下降。现有缩放策略依赖固定锚点、启发式网格或强化学习,缺乏原则性机制来自适应判断何处需细化以及探索多大空间不确定性。我们提出AutoFocus,一种无需训练的、基于不确定性的主动视觉搜索框架。核心洞察是:坐标生成中的分块级困惑度自然反映空间不确定性。AutoFocus不直接输出单一预测,而是采样多个坐标假设,并将其轴向困惑度转换为各向异性高斯空间概率场,显式建模方向性不确定性。基于此场,生成全局与局部区域提案,并引入形状感知缩放以平衡精确定位与上下文保留。最后通过基于视觉提示的聚合步骤,利用结构化比较选择最一致的预测。在ScreenSpot-Pro和ScreenSpot-V2上的大量实验表明,该方法在通用与专用VLM上均实现持续提升。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have enabled autonomous GUI agents that translate natural language instructions into executable screen coordinates. However, grounding performance degrades in high-resolution interfaces, where dense layouts and small interactive elements expose a resolution gap between modern displays and model input constraints. Existing zoom-in strategies rely on fixed anchors, heuristic grids, or reinforcement learning, lacking a principled mechanism to adaptively determine where refinement is needed and how much spatial uncertainty should be explored. We propose AutoFocus, a training-free, uncertainty-aware active visual search framework for GUI grounding. Our key insight is that token-level perplexity in coordinate generation naturally reflects spatial uncertainty. Rather than committing to a single prediction, AutoFocus samples multiple coordinate hypotheses and converts their axial perplexities into an anisotropic gaussian spatial probability field, explicitly modeling directional uncertainty. Based on this field, we generate global and local region proposals and introduce Shape-Aware Zooming to balance tight localization with contextual preservation. A visual prompt-based aggregation step then selects the most consistent prediction via structured comparison. Extensive experiments on ScreenSpot-Pro and ScreenSpot-V2 demonstrate consistent improvements across both general-purpose and GUI-specialized VLMs.

GUI定位视觉语言模型不确定性建模主动搜索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。