让GUI模型知道何时该信任自己，避免误操作。
SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration
- 用分布感知法量化预测不确定性，捕捉输出的空间分散性。
- 测试时校准阈值，保证错误发现率在可控范围内。
- 适合作为高风险自动化系统（如支付）的可靠性保障工具。
图形用户界面（GUI）接地旨在将自然语言指令转化为可执行的屏幕坐标，实现自动化界面交互。然而，错误的接地可能导致代价高昂且难以逆转的操作（如误批准付款），引发对模型可靠性的担忧。本文提出SafeGround，一种面向GUI接地模型的不确定性感知框架，通过测试前校准实现风险感知预测。SafeGround采用分布感知的不确定性量化方法，捕捉任意模型输出中随机样本的空间分散性；随后通过校准过程，推导出具有统计保障的测试时决策阈值，实现错误发现率（FDR）控制。我们在ScreenSpot-Pro这一挑战性基准上对多个GUI接地模型应用SafeGround。实验结果表明，我们的不确定性度量在区分正确与错误预测方面始终优于现有基线，而校准后的阈值能可靠实现严格的风险控制，并具备显著提升系统级准确率的潜力。在多个GUI接地模型上，SafeGround相比仅使用Gemini推理，系统级准确率最高提升5.38个百分点。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) grounding aims to translate natural language instructions into executable screen coordinates, enabling automated GUI interaction. Nevertheless, incorrect grounding can result in costly, hard-to-reverse actions (e.g., erroneous payment approvals), raising concerns about model reliability. In this paper, we introduce SafeGround, an uncertainty-aware framework for GUI grounding models that enables risk-aware predictions through calibrations before testing. SafeGround leverages a distribution-aware uncertainty quantification method to capture the spatial dispersion of stochastic samples from outputs of any given model. Then, through the calibration process, SafeGround derives a test-time decision threshold with statistically guaranteed false discovery rate (FDR) control. We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark. Experimental results show that our uncertainty measure consistently outperforms existing baselines in distinguishing correct from incorrect predictions, while the calibrated threshold reliably enables rigorous risk control and potentials of substantial system-level accuracy improvements. Across multiple GUI grounding models, SafeGround improves system-level accuracy by up to 5.38% percentage points over Gemini-only inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。