arXiv:2508.05615cs.CVcs.AI2025-08AAAI被引 18

通过测试时投票提升界面指令定位准确率,无需额外训练。

Test-Time Reinforcement Learning for GUI Grounding via Region Consistency

  • 测试时生成多预测,用重叠区域找高置信定位点。
  • 不需训练,在ScreenSpot上提升2-3%准确率。
  • 适合资源有限但需高精度的自动化界面操作场景。

图形用户界面(GUI)定位任务旨在将自然语言指令映射到精确屏幕坐标,是自主GUI代理的基础。现有方法依赖大量像素级标注进行监督训练或强化学习,成本高且数据难获取。我们发现,模型对同一界面元素生成多个预测时,空间重叠模式隐含置信度信号,可引导更精准定位。为此提出GUI-RC(区域一致性):在测试时构建空间投票网格,识别模型一致认同的区域。该方法无需训练,即可在ScreenSpot基准上使各类架构准确率提升2-3%。进一步提出GUI-RCPO(区域一致性策略优化),将一致性模式转化为测试时强化学习奖励,通过评估每条预测与集体共识的匹配程度,迭代优化未标注数据上的输出。实验表明,仅用1,272个无标签数据,GUI-RCPO在ScreenSpot上实现3-6%的准确率提升。本方法揭示了测试时扩展与强化学习在GUI定位中的潜力,为更高效的数据利用提供了新路径。

原文摘要 · Abstract (English)

Graphical User Interface (GUI) grounding, the task of mapping natural language instructions to precise screen coordinates, is fundamental to autonomous GUI agents. While existing methods achieve strong performance through extensive supervised training or reinforcement learning with labeled rewards, they remain constrained by the cost and availability of pixel-level annotations. We observe that when models generate multiple predictions for the same GUI element, the spatial overlap patterns reveal implicit confidence signals that can guide more accurate localization. Leveraging this insight, we propose GUI-RC (Region Consistency), a test-time scaling method that constructs spatial voting grids from multiple sampled predictions to identify consensus regions where models show highest agreement. Without any training, GUI-RC improves accuracy by 2-3% across various architectures on ScreenSpot benchmarks. We further introduce GUI-RCPO (Region Consistency Policy Optimization), transforming these consistency patterns into rewards for test-time reinforcement learning. By computing how well each prediction aligns with the collective consensus, GUI-RCPO enables models to iteratively refine their outputs on unlabeled data during inference. Extensive experiments demonstrate the generality of our approach: using only 1,272 unlabeled data, GUI-RCPO achieves 3-6% accuracy improvements across various architectures on ScreenSpot benchmarks. Our approach reveals the untapped potential of test-time scaling and test-time reinforcement learning for GUI grounding, offering a promising path toward more data-efficient GUI agents.

GUI定位测试时学习强化学习无监督

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。