arXiv:2506.03143cs.CLcs.AI2025-06NeurIPS被引 89

无需坐标预测,用注意力机制直接定位界面操作区域。

GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

  • 引入<ACTOR>令牌与视觉片段对齐,一次前向传播生成多个操作区域。
  • 在ScreenSpot-Pro上达44.6分,超越38.1分的UI-TARS-72B模型。
  • 仅微调1亿参数的动作头即可媲美顶尖模型,保持主干模型通用性。

构建视觉语言模型(VLM)驱动的GUI智能体面临的核心挑战是视觉定位,即根据视觉内容和文本计划准确找到执行动作的屏幕区域。现有方法多将此问题建模为基于文本的坐标生成任务,但存在空间语义对齐弱、无法处理模糊标注目标、以及屏幕坐标密集性与视觉特征粗粒度不匹配等问题。本文提出GUI-Actor,一种基于VLM的无坐标视觉定位方法。其核心是设计一个基于注意力的动作头,使专用的<ACTOR>令牌与所有相关视觉片段对齐,实现单次前向传播生成一个或多个动作区域。同时,我们进一步构建了定位验证器,从候选区域中评估并选择最合理的动作区域。大量实验表明,GUI-Actor在多个GUI动作定位基准上优于现有最先进方法,对未见过的屏幕分辨率和布局具有更强泛化能力。值得注意的是,以Qwen2.5-VL为骨干的GUI-Actor-7B在ScreenSpot-Pro上达到44.6分,超过UI-TARS-72B的38.1分。此外,通过引入验证器,仅微调新提出的动作头(7B模型约1亿参数)而冻结VLM主干,即可实现与先前最先进模型相当的性能,证明GUI-Actor可在不损害主干模型通用能力的前提下,高效赋予其强大的定位能力。

原文摘要 · Abstract (English)

One of the principal challenges in building VLM-powered GUI agents is visual grounding, i.e., localizing the appropriate screen region for action execution based on both the visual content and the textual plans. Most existing work formulates this as a text-based coordinate generation task. However, these approaches suffer from several limitations: weak spatial-semantic alignment, inability to handle ambiguous supervision targets, and a mismatch between the dense nature of screen coordinates and the coarse, patch-level granularity of visual features extracted by models like Vision Transformers. In this paper, we propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its core, GUI-Actor introduces an attention-based action head that learns to align a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the model to propose one or more action regions in a single forward pass. In line with this, we further design a grounding verifier to evaluate and select the most plausible action region from the candidates proposed for action execution. Extensive experiments show that GUI-Actor outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks, with improved generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7 with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by incorporating the verifier, we find that fine-tuning only the newly introduced action head (~100M parameters for 7B model) while keeping the VLM backbone frozen is sufficient to achieve performance comparable to previous state-of-the-art models, highlighting that GUI-Actor can endow the underlying VLM with effective grounding capabilities without compromising its general-purpose strengths.

视觉定位GUI智能体注意力机制VLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。