通过抑制背景与聚焦中心,提升界面元素定位精度。
V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peaking
- 引入抑制注意力机制,减少背景干扰
- 基于费茨定律建模高斯热图,区分中心与边缘
- 在两个基准上达到92.4%和52.5%准确率
精确的GUI元素定位对GUI代理开发至关重要。传统方法依赖边界框或中心点回归,忽略空间交互不确定性与视觉-语义层次结构。近期方法虽引入注意力机制,但仍存在两大问题:(1) 忽略背景区域处理导致注意力漂移;(2) 对目标元素统一建模,无法区分中心与边缘,造成点击不精准。受人类视觉处理界面方式启发,我们提出谷到峰(V2P)方法。为缓解背景干扰,V2P引入抑制注意力机制,最小化模型对无关区域的关注,突出目标区域。针对中心-边缘区分问题,V2P采用费茨定律启发式方法,将界面交互建模为二维高斯热图,权重从中心向边缘递减,方差由目标尺寸决定。因此,V2P有效隔离目标区域,并引导模型聚焦于界面元素最关键位置。使用V2P训练的模型在ScreenSpot-v2和ScreenSpot-Pro两个基准上分别达到92.4%和52.5%的性能(见图~\ref{fig:main_results_charts})。消融实验进一步验证各组件贡献,凸显V2P在精确GUI定位任务中的通用性及其在未来GUI代理实际部署中的潜力。
原文摘要 · Abstract (English)
Precise localization of GUI elements is crucial for the development of GUI agents. Traditional methods rely on bounding box or center-point regression, neglecting spatial interaction uncertainty and visual-semantic hierarchies. Recent methods incorporate attention mechanisms but still face two key issues: (1) ignoring processing background regions causes attention drift from the desired area, and (2) uniform modeling the target UI element fails to distinguish between its center and edges, leading to click imprecision. Inspired by how humans visually process and interact with GUI elements, we propose the Valley-to-Peak (V2P) method to address these issues. To mitigate background distractions, V2P introduces a suppression attention mechanism that minimizes the model's focus on irrelevant regions to highlight the intended region. For the issue of center-edge distinction, V2P applies a Fitts' Law-inspired approach by modeling GUI interactions as 2D Gaussian heatmaps where the weight gradually decreases from the center towards the edges. The weight distribution follows a Gaussian function, with the variance determined by the target's size. Consequently, V2P effectively isolates the target area and teaches the model to concentrate on the most essential point of the UI element. The model trained by V2P achieves the performance with 92.4\% and 52.5\% on two benchmarks ScreenSpot-v2 and ScreenSpot-Pro (see Fig.~\ref{fig:main_results_charts}). Ablations further confirm each component's contribution, underscoring V2P's generalizability in precise GUI grounding tasks and its potential for real-world deployment in future GUI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。