arXiv:2507.15846cs.LGcs.AI2025-07AAAI被引 58

用高斯分布建模点击位置,让AI更精准地理解图形界面操作。

GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding

  • 将界面元素建模为高斯分布,实现连续空间奖励。
  • 在ScreenSpot-Pro上比最强基线提升24.7%。
  • 适合需要精确交互的自动化界面任务研究者。

图形用户界面(GUI)定位任务旨在将自然语言指令映射到精确的界面位置以实现自主交互。现有强化学习方法使用二值奖励,将界面元素视为非黑即白的目标,产生稀疏信号,忽略了空间交互的连续性。受人类点击行为自然呈现以目标为中心的高斯分布启发,我们提出GUI-G$^2$:一种基于高斯奖励的界面定位框架,将界面元素建模为界面上连续的高斯分布。该框架包含两种协同机制:高斯点奖励通过以元素中心为峰值、指数衰减的分布实现精确定位;覆盖奖励通过测量预测高斯分布与目标区域的重叠度评估空间对齐。为应对不同元素尺度,设计自适应方差机制,根据元素尺寸动态校准奖励分布。该框架将GUI定位从稀疏的二值分类转变为密集的连续优化,高斯分布生成丰富的梯度信号,引导模型逼近最优交互位置。在ScreenSpot、ScreenSpot-v2和ScreenSpot-Pro基准上的大量实验表明,GUI-G$^2$显著优于当前最优方法UI-TARS-72B,尤其在ScreenSpot-Pro上提升达24.7%。分析显示,连续建模对界面变化更具鲁棒性,并能更好泛化至未见布局,确立了GUI交互中空间推理的新范式。

原文摘要 · Abstract (English)

Graphical User Interface (GUI) grounding maps natural language instructions to precise interface locations for autonomous interaction. Current reinforcement learning approaches use binary rewards that treat elements as hit-or-miss targets, creating sparse signals that ignore the continuous nature of spatial interactions. Motivated by human clicking behavior that naturally forms Gaussian distributions centered on target elements, we introduce GUI Gaussian Grounding Rewards (GUI-G$^2$), a principled reward framework that models GUI elements as continuous Gaussian distributions across the interface plane. GUI-G$^2$ incorporates two synergistic mechanisms: Gaussian point rewards model precise localization through exponentially decaying distributions centered on element centroids, while coverage rewards assess spatial alignment by measuring the overlap between predicted Gaussian distributions and target regions. To handle diverse element scales, we develop an adaptive variance mechanism that calibrates reward distributions based on element dimensions. This framework transforms GUI grounding from sparse binary classification to dense continuous optimization, where Gaussian distributions generate rich gradient signals that guide models toward optimal interaction positions. Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro benchmarks demonstrate that GUI-G$^2$, substantially outperforms state-of-the-art method UI-TARS-72B, with the most significant improvement of 24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides superior robustness to interface variations and enhanced generalization to unseen layouts, establishing a new paradigm for spatial reasoning in GUI interaction tasks.

GUI定位高斯奖励强化学习人机交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。