一个轻量级视觉-语言-动作模型，让AI像人一样看懂并操作界面。
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
- 用界面结构引导视觉标记选择，减少33%冗余计算。
- 零样本截图定位准确率达75.1%，推理速度提升1.4倍。
- 适合做网页/手机端自动化任务的开发者或研究者。
构建图形用户界面（GUI）助手有望显著提升人类工作流效率。现有多数代理依赖封闭源代码API与文本丰富的元信息（如HTML或可访问性树），难以像人类一样感知界面视觉，凸显了开发GUI视觉代理的必要性。本文提出一种数字世界中的视觉-语言-动作模型ShowUI，其创新包括：(i) UI引导的视觉标记选择，将截图建模为界面连接图，自适应识别冗余关系，作为自注意力模块中标记选择的标准，降低计算成本；(ii) 视觉-语言-动作交错流，灵活统一多种GUI任务需求，有效管理导航中的视觉-动作历史，并对每张截图配对多轮查询-动作序列，提升训练效率；(iii) 通过精心数据筛选与重采样策略构建小规模高质量指令跟随数据集，缓解数据类型失衡问题。基于上述组件，仅使用256K数据的轻量级2B模型在零样本截图定位任务中达到75.1%准确率；其UI引导的标记选择在训练中减少33%冗余视觉标记，使性能提速1.4倍。跨网页Mind2Web、移动端AITW及在线MiniWob环境的导航实验进一步验证了该模型在推进GUI视觉代理方面的有效性。模型开源地址：https://github.com/showlab/ShowUI。
原文摘要 · Abstract (English)
Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. While most agents are language-based, relying on closed-source API with text-rich meta-information (e.g., HTML or accessibility tree), they show limitations in perceiving UI visuals as humans do, highlighting the need for GUI visual agents. In this work, we develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations: (i) UI-Guided Visual Token Selection to reduce computational costs by formulating screenshots as an UI connected graph, adaptively identifying their redundant relationship and serve as the criteria for token selection during self-attention blocks; (ii) Interleaved Vision-Language-Action Streaming that flexibly unifies diverse needs within GUI tasks, enabling effective management of visual-action history in navigation or pairing multi-turn query-action sequences per screenshot to enhance training efficiency; (iii) Small-scale High-quality GUI Instruction-following Datasets by careful data curation and employing a resampling strategy to address significant data type imbalances. With above components, ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection further reduces 33% of redundant visual tokens during training and speeds up the performance by 1.4x. Navigation experiments across web Mind2Web, mobile AITW, and online MiniWob environments further underscore the effectiveness and potential of our model in advancing GUI visual agents. The models are available at https://github.com/showlab/ShowUI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。