arXiv:2604.23941cs.CV2026-04

轻量级GUI元素定位模型,手机端也能高效运行

GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction

论文配图:GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
图 1 · 摘自论文原文
  • 采用编码器-解码器架构,小模型效果优于传统缩小版大模型
  • 通过数据精炼从1080万样本中筛选出380万高质量数据训练
  • 仅2.3亿参数却达到大模型精度,适合移动端部署

图形用户界面(GUI)元素定位(基于自然语言指令精准定位截图中的元素)是实现自主GUI交互的基础。在移动等资源受限设备上直接部署该能力,对低延迟的GUI代理日益重要。然而,现有视觉定位方法普遍使用超大规模视觉语言模型(超过25亿参数),因内存和计算限制难以在设备端运行。为此,本文提出GoClick,一个仅含2.3亿参数的轻量级GUI元素定位视觉语言模型,其定位准确率可媲美远大于它的模型。单纯缩小现有仅解码器架构的VLM无法获得理想效果,我们改用编码器-解码器结构,在小规模参数下表现更优。此外,为弥补小模型容量不足,我们设计了渐进式数据精炼流程,通过任务类型过滤与数据比例调整,从1080万原始样本中提取出380万高质量核心数据集。使用该数据集训练的GoClick在多个基准测试中表现优异,模型体积小、推理速度快。将其集成到设备-云端协同框架后,还能提升云端任务规划器的元素定位精度与任务成功率。本工作为GUI代理领域提供了一种可行的轻量化探索路径。

原文摘要 · Abstract (English)

Graphical User Interface (GUI) element grounding (precisely locating elements on screenshots based on natural language instructions) is fundamental for agents interacting with GUIs. Deploying this capability directly on resource-constrained devices like mobile phones is increasingly critical for GUI agents requiring low latency. However, this goal faces a significant challenge, as current visual grounding methods typically employ large vision-language model (VLM) (more than 2.5B parameters), making them impractical for on-device execution due to memory and computational constraints. To address this, this paper introduces GoClick, a lightweight GUI element grounding VLM with only 230M parameters that achieves excellent visual grounding accuracy, even on par with significantly larger models. Simply downsizing existing decoder-only VLMs is a straightforward way to design a lightweight model, but our experiments reveal that this approach yields suboptimal results. Instead, we select an encoder-decoder architecture, which outperforms decoder-only alternatives at small parameter scales for GUI grounding tasks. Additionally, the limited capacity of small VLMs encourages us to develop a Progressive Data Refinement pipeline that utilizes task type filtering and data ratio adjustment to extract a high-quality 3.8M-sample core set from a 10.8M raw dataset. Training GoClick using this core set brings notable grounding accuracy gains. Our experiments show that GoClick excels on multiple GUI element grounding benchmarks while maintaining a small size and high inference speed. GoClick also enhances GUI agent performance when integrated into a device-cloud collaboration framework, where GoClick helps cloud-based task planners perform precise element localization and achieve higher success rates. We hope our method serves as a meaningful exploration within the GUI agent community.

GUI定位轻量化模型移动部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。