根据模型不确定度动态调整界面元素放大区域，提升小图标定位准确率。
UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
- 基于置信度与空间一致性判断是否需要放大，仅在不确定时触发
- 按实例差异自动计算最佳放大范围，最高提升13.4%定位准确率
- 无需训练即可适配多种模型，适合密集界面场景的精准定位
GUI定位任务中，小图标和密集布局导致定位困难。现有测试时放大方法对所有实例采用固定大小统一裁剪，忽略模型实际不确定性。本文提出无需训练的自适应放大框架UI-Zoomer，将触发时机与放大尺度建模为不确定性量化问题。通过融合随机候选的空间一致性与标记级生成置信度，构建置信度感知门控机制，仅在定位不确定时触发放大。触发后，基于总方差定律分解预测方差为样本间位置发散与样本内框体范围，实现每实例个性化裁剪半径。在ScreenSpot-Pro、UI-Vision和ScreenSpot-v2上实验表明，该方法在多种模型架构下均优于强基线，分别取得+13.4%、+10.3%和+4.2%的提升，且无需额外训练。
原文摘要 · Abstract (English)
GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose \textbf{UI-Zoomer}, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4\%, +10.3\%, and +4.2\% respectively, with no additional training required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。