arXiv:2601.03928cs.CVcs.AI2026-01被引 7

通过保留位置信息的视觉标记选择,让AI更高效精准地理解界面操作。

FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selection

  • 基于指令和界面结构特征筛选关键视觉区块,减少冗余计算。
  • 仅保留30%视觉标记仍保持96.8%性能,推理速度提升1.44倍。
  • 适合需要快速响应的UI自动化与智能助手场景。

视觉语言模型在用户界面(UI)定位任务中表现优异,得益于其处理高分辨率截图的能力。然而,2K分辨率的截图会生成约4700个视觉标记,带来显著计算开销并稀释注意力。人类在操作界面时通常聚焦于兴趣区域。本文首次提出高效UI定位任务,提出FocusUI框架:通过融合指令相关评分与基于规则的界面图评分,抑制大块同质区域,选出与指令相关的视觉标记;同时引入新策略PosPad,将被剔除的连续视觉标记压缩为单个特殊标记并置于序列末尾,以保持位置连续性。在四个基准测试上的实验表明,FocusUI优于专用基线。在ScreenSpot-Pro上,FocusUI-7B相比GUI-Actor-7B提升3.7%;即使仅保留30%视觉标记,性能仅下降3.2%,推理速度最高提升1.44倍,峰值显存降低17%。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have shown remarkable performance in User Interface (UI) grounding tasks, driven by their ability to process increasingly high-resolution screenshots. However, screenshots are tokenized into thousands of visual tokens (e.g., about 4700 for 2K resolution), incurring significant computational overhead and diluting attention. In contrast, humans typically focus on regions of interest when interacting with UI. In this work, we pioneer the task of efficient UI grounding. Guided by practical analysis of the task's characteristics and challenges, we propose FocusUI, an efficient UI grounding framework that selects patches most relevant to the instruction while preserving positional continuity for precise grounding. FocusUI addresses two key challenges: (1) Eliminating redundant tokens in visual encoding. We construct patch-level supervision by fusing an instruction-conditioned score with a rule-based UI-graph score that down-weights large homogeneous regions to select distinct and instruction-relevant visual tokens. (2) Preserving positional continuity during visual token selection. We find that general visual token pruning methods suffer from severe accuracy degradation on UI grounding tasks due to broken positional information. We introduce a novel PosPad strategy, which compresses each contiguous sequence of dropped visual tokens into a single special marker placed at the sequence's last index to preserve positional continuity. Comprehensive experiments on four grounding benchmarks demonstrate that FocusUI surpasses GUI-specific baselines. On the ScreenSpot-Pro benchmark, FocusUI-7B achieves a performance improvement of 3.7% over GUI-Actor-7B. Even with only 30% visual token retention, FocusUI-7B drops by only 3.2% while achieving up to 1.44x faster inference and 17% lower peak GPU memory.

UI理解视觉编码效率优化定位任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。