用空间推理和搜索提升GUI定位效率,少数据也能高精度。
ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search
- 自生成推理过程+空间先验批评,提升模型自我纠错能力。
- 仅用0.2%训练数据,性能超越现有开源基线。
- 适合低资源场景下的自动化界面交互研究者。
多模态大语言模型(MLLMs)已能通过图形用户界面(GUI)与计算机交互,但精确定位界面元素(如按钮)坐标仍具挑战性,以往工作依赖大规模网页数据集提升定位准确率。本文提出ReGUIDE框架,通过自生成推理和空间感知批评实现高效学习。具体而言,ReGUIDE利用在线强化学习自动生成语言推理过程以定位目标,并通过空间先验对预测进行批评,强制模型在输入变换下保持等变性。推理时,进一步采用测试时缩放策略,结合空间搜索与坐标聚合提升性能。实验表明,ReGUIDE在多个基准上显著优于基线方法,仅需0.2%的训练样本即达到领先效果。
原文摘要 · Abstract (English)
Recent advances in Multimodal Large Language Models (MLLMs) have enabled autonomous agents to interact with computers via Graphical User Interfaces (GUIs), where accurately localizing the coordinates of interface elements (e.g., buttons) is often required for fine-grained actions. However, this remains significantly challenging, leading prior works to rely on large-scale web datasets to improve the grounding accuracy. In this work, we propose Reasoning Graphical User Interface Grounding for Data Efficiency (ReGUIDE), a novel and effective framework for web grounding that enables MLLMs to learn data efficiently through self-generated reasoning and spatial-aware criticism. More specifically, ReGUIDE learns to (i) self-generate a language reasoning process for the localization via online reinforcement learning, and (ii) criticize the prediction using spatial priors that enforce equivariance under input transformations. At inference time, ReGUIDE further boosts performance through a test-time scaling strategy, which combines spatial search with coordinate aggregation. Our experiments demonstrate that ReGUIDE significantly advances web grounding performance across multiple benchmarks, outperforming baselines with substantially fewer training data points (e.g., only 0.2% samples compared to the best open-sourced baselines).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。