用强化学习让模型自己评自己,提升图形界面定位精度。
Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding

- 让模型先提方案,再自评优劣,动态优化定位结果。
- 在6个基准上定位准确率显著提升,自评可靠性更强。
- 适合需要高精度界面操作的自动化系统开发者。
图形用户界面(GUI)定位需将自然语言指令映射到精确像素坐标。由于视觉元素相似且布局密集,模型虽能理解语义却难以精确定位。尽管增加采样次数(Pass@k)有潜力提升性能,但基于几何聚类的静态一致性策略改善有限,因预测点常分散。本文提出以可学习的选择机制替代静态策略,通过在截图上渲染自身提案来批判选择最优目标。针对模型提议与评判能力的巨大差距,设计协同演化的“提议-批评”框架。为联合优化,引入成熟度感知的自适应协同进化强化学习范式,动态平衡两者的训练目标:提议者输出多样性增强批评者鲁棒性,批评者成熟判别力则释放提议者更广空间探索潜力,形成双向促进与协同进化,确保对多样复杂界面布局的泛化能力。在6个基准上的大量实验表明,该方法显著提升定位准确率与批评可靠性。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) grounding requires mapping natural language instructions to precise pixel coordinates. However, due to visually homogeneous elements and dense layouts, models typically grasp semantic intent yet struggle with achieving precise localization. While scaling sampling attempts (Pass@k) reveals potential gains, static self-consistency strategies derived from geometric clustering often yield limited improvements, as the model's predictions tend to be spatially dispersed. In this paper, we propose replacing static consistency strategies with a learnable selection mechanism that selects the optimal target by critiquing its own proposals rendered on the screenshot. Given the significant disparity between the model's grounding and critiquing capabilities, we propose a co-evolving Propose-then-Critic framework. To jointly optimize these, we introduce a maturity-aware adaptive co-evolutionary reinforcement learning paradigm. This approach dynamically balances the training objectives of proposer and critic, where the diversity of the proposer's outputs enhances critic robustness, while the critic's maturing discrimination capability conversely unlocks the proposer's potential for extensive spatial exploration, fostering the mutual reinforcement and co-evolution of both capabilities, thereby ensuring generalizability to adapt to diverse and complex interface layouts. Extensive experiments over 6 benchmarks show that our method significantly enhances both grounding accuracy and critic reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。