纯视觉控制电脑界面，定位精准效率高。
Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
- 分治架构：通用大模型解读指令，专用模型精确定位元素
- 在ScreenSpot基准上定位准确率提升22.5%（绝对领先）
- 跨平台适用，适合自动化测试与辅助工具开发
现有GUI代理多依赖非视觉输入（如HTML源码或可访问性树），限制了其在多样化软件环境中的灵活性。当前多模态大语言模型虽擅长视觉定位真实物体，但在GUI元素定位上仍因现实物体与界面元素间的语义鸿沟而表现不佳。本文提出Ponder & Press框架，仅使用视觉输入实现通用计算机控制。该框架采用分治策略：通用多模态大模型作为‘解释器’，将用户高层指令转化为详细动作描述；专用GUI多模态大模型作为‘定位器’，精确识别动作目标位置。通过纯视觉输入，该代理展现出类人交互能力，适用于广泛应用场景。在ScreenSpot GUI定位基准上，其定位器性能优于现有模型22.5%。在多种GUI环境（网页、桌面软件、移动UI）的离线与交互式评测中，该框架均达到当前最优表现，验证了纯视觉GUI代理的巨大潜力。
原文摘要 · Abstract (English)
Most existing GUI agents typically depend on non-vision inputs like HTML source code or accessibility trees, limiting their flexibility across diverse software environments and platforms. Current multimodal large language models (MLLMs), which excel at using vision to ground real-world objects, offer a potential alternative. However, they often struggle with accurately localizing GUI elements -- a critical requirement for effective GUI automation -- due to the semantic gap between real-world objects and GUI elements. In this work, we introduce Ponder & Press, a divide-and-conquer framework for general computer control using only visual input. Our approach combines an general-purpose MLLM as an 'interpreter', responsible for translating high-level user instructions into detailed action descriptions, with a GUI-specific MLLM as a 'locator' that precisely locates GUI elements for action placement. By leveraging a purely visual input, our agent offers a versatile, human-like interaction paradigm applicable to a wide range of applications. Ponder & Press locator outperforms existing models by +22.5% on the ScreenSpot GUI grounding benchmark. Both offline and interactive agent benchmarks across various GUI environments -- including web pages, desktop software, and mobile UIs -- demonstrate that Ponder & Press framework achieves state-of-the-art performance, highlighting the potential of visual GUI agents. Refer to the project homepage https://invinciblewyq.github.io/ponder-press-page/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。