让AI理解GUI操作指令，纯视觉实现精准定位。
Aria-UI: Visual Grounding for GUI Instructions
- 纯视觉方法，不依赖HTML或AXTree等辅助输入。
- 在离线与在线基准上均超越现有模型表现。
- 适合研究GUI自动化、多模态交互的开发者使用。
通过直接操控GUI来自动化跨平台任务的数字代理日益重要。然而，由于依赖HTML或AXTree输入，将语言指令准确映射到目标元素仍是重大挑战。本文提出Aria-UI，一个专为GUI接地设计的大规模多模态模型。Aria-UI采用纯视觉方法，摒弃对辅助输入的依赖。为适应多样化的规划指令，我们构建了可扩展的数据流水线，合成丰富且高质量的接地指令样本。为应对任务执行中的动态上下文，Aria-UI引入文本与图文交错的动作历史，实现鲁棒的上下文感知推理。Aria-UI在离线与在线代理基准上均达到新最优性能，显著优于仅视觉或依赖AXTree的基线模型。所有训练数据与模型检查点已开源，网址：https://ariaui.github.io。
原文摘要 · Abstract (English)
Digital agents for automating tasks across different platforms by directly manipulating the GUIs are increasingly important. For these agents, grounding from language instructions to target elements remains a significant challenge due to reliance on HTML or AXTree inputs. In this paper, we introduce Aria-UI, a large multimodal model specifically designed for GUI grounding. Aria-UI adopts a pure-vision approach, eschewing reliance on auxiliary inputs. To adapt to heterogeneous planning instructions, we propose a scalable data pipeline that synthesizes diverse and high-quality instruction samples for grounding. To handle dynamic contexts in task performing, Aria-UI incorporates textual and text-image interleaved action histories, enabling robust context-aware reasoning for grounding. Aria-UI sets new state-of-the-art results across offline and online agent benchmarks, outperforming both vision-only and AXTree-reliant baselines. We release all training data and model checkpoints to foster further research at https://ariaui.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。