让AI操作界面更可信，能自检判断是否该动手。
Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning
- 用自我批判式强化学习，让AI同时预测点击位置和信心值。
- 在多个测试集上保持高准确率，且信心评估更贴近真实正确性。
- 适合需要可靠决策的自动化场景，如智能客服或机器人操作。
自主图形用户界面（GUI）代理依赖于准确的GUI定位，即将语言指令映射到屏幕坐标以执行用户命令。然而，当前模型无论是通过监督微调（SFT）还是强化学习（RL）训练，其提供的置信度信号往往与实际定位准确性不一致，导致过度自信且不可靠的预测。为此，我们提出HyperClick，一种通过自批判强化学习（SCRL）提升可信GUI定位的新框架。HyperClick结合正确性奖励与置信度对齐奖励，训练策略模型输出点击预测和显式的置信度估计。该方法通过基于置信度的自我评估，联合优化定位准确性和置信度可靠性。在多个挑战性基准上的大量实验表明，HyperClick在保持强大定位性能的同时，提供了更对齐的置信度估计。通过在GUI操作中暴露不确定性，HyperClick支持基于置信度的放弃执行机制。代码将在此处发布。
原文摘要 · Abstract (English)
Autonomous graphical user interface (GUI) agents rely on accurate GUI grounding, which maps language instructions to on-screen coordinates, to execute user commands. However, current models, whether trained via supervised fine-tuning (SFT) or reinforcement learning (RL), often provide confidence signals that are poorly aligned with actual grounding correctness, leading to overconfident and unreliable predictions. To address this, we propose HyperClick, a novel framework that enhances trustworthy GUI grounding through self-critiqued reinforcement learning (SCRL). HyperClick combines a correctness reward and a confidence alignment reward, training the policy model to output both a click prediction and an explicit confidence estimate. This approach jointly optimizes grounding accuracy and confidence reliability through confidence-based self-assessment. Extensive experiments on challenging benchmarks show that HyperClick maintains strong grounding performance while providing better-aligned confidence estimates. By exposing uncertainty alongside GUI actions, HyperClick supports confidence-based abstention in GUI automation. Code will be released here.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。