通过多视角一致性训练提升GUI定位准确率
VISTA: View-Consistent Self-Verified Training for GUI Grounding
- 从同一界面生成多个保留目标元素的视角进行对比学习
- 在ScreenSpot-Pro上将模型准确率从55.5%提升至63.4%
- 适合需要高鲁棒性GUI理解的智能助手开发
在应用组相对策略优化（GRPO）进行GUI定位时，样本仅来自单一截图视角；导致在困难实例上所有组别均为失败，或在简单实例上全部成功，无法获得有效相对优势。本文提出VISTA（视图一致自验证训练），一种基于GRPO的训练框架，将每个比较组构建为同一GUI实例的多个目标保持可见的视角。每个视角由保持目标元素可见并精确重映其框体的裁剪生成，使模型推断在语义等价但几何不同的输入间进行对比。为稳定短坐标生成过程，防止强化学习退化为无条件模仿，VISTA引入自验证跨视角锚点：一个以优势加权损失优化的最优答案，不参与组基线，仅在模型生成最高奖励推断时激活。在五个GUI定位基准测试及多个Qwen骨干模型上，VISTA持续提升定位精度。在ScreenSpot-Pro上，Qwen3-VL 4B/8B/30B-A3B准确率分别从55.5%/52.7%/53.7%提升至63.4%/65.8%/67.0%。鲁棒性分析显示，其最差视角准确率更高，预测翻转率更低。
原文摘要 · Abstract (English)
When applying Group Relative Policy Optimization (GRPO) for GUI Grounding, rollouts are sampled from a single screenshot view; groups often become either all failures on difficult instances or all successes on easy ones, yielding no useful relative advantage. We propose VISTA (View-Consistent Self-Verified Training), a GRPO-based training framework that constructs each comparison group from multiple target-preserving views of the same GUI instance.Each view is generated by a crop that keeps the target element visible and remaps its box exactly, so model rollouts are compared across semantically equivalent but geometrically different inputs. To stabilize short coordinate generation without turning reinforcement learning into unconditional imitation, VISTA further adds a self-verified cross-view anchor: an oracle answer optimized with an advantage-weighted loss, excluded from the group baseline and activated only when the model has produced a maximum-reward rollout. Across five GUI-grounding benchmarks and multiple Qwen backbones, VISTA consistently improves grounding accuracy.On ScreenSpot-Pro, it raises Qwen3-VL 4B/8B/30B-A3B from 55.5/52.7/53.7 to 63.4/65.8/67.0. Robustness analyses further show higher worst-view accuracy and lower prediction flip rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。