无需训练即可提升界面识别模型准确率，解决高分辨率与复杂元素带来的偏差。
BAMI: Training-Free Bias Mitigation in GUI Grounding
- 通过粗到精聚焦和候选选择策略，主动消除界面识别中的偏差
- 在ScreenSpot-Pro上使TianXi-Action-7B模型准确率从51.9%提升至57.8%
- 完全无需微调，适合作为通用增强工具集成到各类GUI模型中
GUI接地是实现点击、拖拽等任务的关键能力。但在ScreenSpot-Pro等复杂场景下，现有模型表现不佳。我们提出掩码预测分布（MPD）归因方法，发现误差主要源于高图像分辨率（精度偏差）和复杂界面元素（模糊性偏差）。为此，我们设计了无训练偏置感知操作推理（BAMI），引入粗到精聚焦和候选选择两种操作，有效缓解上述偏差。大量实验表明，BAMI在无训练设置下显著提升多种GUI接地模型的准确性。例如，应用于TianXi-Action-7B模型后，其在ScreenSpot-Pro上的准确率从51.9%提升至57.8%。消融实验进一步验证了BAMI在不同参数配置下的鲁棒性与稳定性。代码已开源：https://github.com/Neur-IO/BAMI。
原文摘要 · Abstract (English)
GUI grounding is a critical capability for enabling GUI agents to execute tasks such as clicking and dragging. However, in complex scenarios like the ScreenSpot-Pro benchmark, existing models often suffer from suboptimal performance. Utilizing the proposed \textbf{Masked Prediction Distribution (MPD)} attribution method, we identify that the primary sources of errors are twofold: high image resolution (leading to precision bias) and intricate interface elements (resulting in ambiguity bias). To address these challenges, we introduce \textbf{Bias-Aware Manipulation Inference (BAMI)}, which incorporates two key manipulations, coarse-to-fine focus and candidate selection, to effectively mitigate these biases. Our extensive experimental results demonstrate that BAMI significantly enhances the accuracy of various GUI grounding models in a training-free setting. For instance, applying our method to the TianXi-Action-7B model boosts its accuracy on the ScreenSpot-Pro benchmark from 51.9\% to 57.8\%. Furthermore, ablation studies confirm the robustness of the BAMI approach across diverse parameter configurations, highlighting its stability and effectiveness. Code is available at https://github.com/Neur-IO/BAMI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。