用截图实现Windows界面精准操作，让AI懂用户指令并点对按钮。
WinClick: GUI Grounding with Multimodal Large Language Models
- 用截图+大模型识别可点击区域，不依赖网页结构数据。
- 预训练提升界面元素定位准确率，实测性能超越现有方法。
- 专为Windows设计的首个基准测试集，适合桌面自动化研究者。
图形用户界面（GUI）任务对自动化软件测试、界面导航等工作至关重要。以往研究指出，构建视觉化GUI代理的关键挑战在于GUI接地——根据指令精准定位屏幕元素。然而，多数现有代理依赖训练或推理时的结构化数据（如DOM或HTML文件），在通用桌面环境（如Windows系统）中难以获取。为此，我们提出WinClick，一种基于Windows平台的新型视觉化GUI代理。WinClick利用截图检测可操作区域，并通过引入GUI接地预训练及基于大语言模型的对齐方法，解决接地难题。此外，我们构建了首个针对Windows平台的综合性基准测试集WinSpot。实验表明，结合接地预训练的WinClick显著优于现有基线，为桌面环境下的GUI自动化提供可扩展解决方案。WinSpot已公开于https://github.com/zackhuiiiii/WinSpot。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) tasks are vital for automating workflows such as software testing, user interface navigation. For users, the GUI is the most intuitive platform for interacting with a computer. Previous work identified a key challenge in developing visual GUI agents: GUI grounding - the ability to accurately locate screen elements based on instructions. However, most existing GUI agents rely on structured data formats like DOM or HTML files in training or inferencing, which are inaccessible across all applications, particular in a general desktop environments such as Windows OS. To address this, we introduce WinClick, a novel visual GUI agent developed in Windows platform. WinClick leverages screenshots to detect actionable regions. To overcome the challenge of GUI grounding, we enhance WinClick with GUI grounding pre-training and propose an LLM-based method for aligning GUI grounding data. Additionally, we introduce WinSpot, the first comprehensive benchmark for GUI grounding on Windows. Our experiments demonstrate that WinClick, combined with GUI grounding pre-training, significantly outperforms existing baselines, offering a scalable solution for GUI automation in desktop environments. WinSpot is publicly available at https://github.com/zackhuiiiii/WinSpot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。