通过多视角预测提升界面指令定位的稳定性与精度
MVP: Multiple View Prediction Improves GUI Grounding
- 利用注意力引导生成多个裁剪视角,融合多视图预测结果
- 在ScreenSpot-Pro上使最大模型准确率提升至74.0%
- 无需训练即可增强现有模型性能,适合高分辨率小元素场景
GUI定位将自然语言指令转化为精确像素坐标,是构建实用GUI代理的关键。然而,现有模型存在显著坐标预测不稳定性,微小视觉扰动(如裁剪几个像素)即可能彻底改变预测结果,导致正确与错误之间切换。这种不稳定性严重制约了模型在高分辨率及小型UI元素样本上的表现。为此,我们提出无需训练的多视角预测(MVP)框架,通过多视角推理提升定位性能。核心思想是:虽然单视角预测可能不稳定,但聚合多个精心裁剪视角的预测可有效区分正确坐标与异常值。MVP包含两个组件:(1) 注意力引导视角生成,基于指令-图像注意力分数生成多样化视图;(2) 多坐标聚类,通过选取空间密度最高的聚类中心来集成预测。大量实验表明,MVP在多种模型和基准上均具有效性。尤其在ScreenSpot-Pro上,使UI-TARS-1.5-7B达到56.1%,GTA1-7B达61.7%,Qwen3VL-8B-Instruct达65.3%,Qwen3VL-32B-Instruct达74.0%。代码已开源。
原文摘要 · Abstract (English)
GUI grounding, which translates natural language instructions into precise pixel coordinates, is essential for developing practical GUI agents. However, we observe that existing grounding models exhibit significant coordinate prediction instability, minor visual perturbations (e.g. cropping a few pixels) can drastically alter predictions, flipping results between correct and incorrect. This instability severely undermines model performance, especially for samples with high-resolution and small UI elements. To address this issue, we propose Multi-View Prediction (MVP), a training-free framework that enhances grounding performance through multi-view inference. Our key insight is that while single-view predictions may be unstable, aggregating predictions from multiple carefully cropped views can effectively distinguish correct coordinates from outliers. MVP comprises two components: (1) Attention-Guided View Proposal, which derives diverse views guided by instruction-to-image attention scores, and (2) Multi-Coordinates Clustering, which ensembles predictions by selecting the centroid of the densest spatial cluster. Extensive experiments demonstrate MVP's effectiveness across various models and benchmarks. Notably, on ScreenSpot-Pro, MVP boosts UI-TARS-1.5-7B to 56.1%, GTA1-7B to 61.7%, Qwen3VL-8B-Instruct to 65.3%, and Qwen3VL-32B-Instruct to 74.0%. The code is available at https://github.com/ZJUSCL/MVP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。