通过自适应缩放与指令优化,提升复杂界面元素定位准确率
AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
- 引入指令优化模块,将模糊指令转为明确描述
- 采用条件缩放策略,仅对小元素二次推理,提升精度并节省计算
- 在公开基准上表现领先,适合高分辨率界面智能代理应用
GUI 基础定位是视觉语言模型实现自动化人机交互的关键能力,旨在从自然语言指令中定位图形用户界面的目标元素。然而,由于图像分辨率高、界面元素小且指令模糊,该任务仍具挑战性。本文提出 AdaZoom-GUI,一种基于自适应缩放的 GUI 定位框架,同时提升定位精度与指令理解能力。方法包括:一个指令优化模块,将自然语言命令重写为更明确、详细的描述;一种条件缩放策略,仅对预测的小元素进行第二阶段推理,避免简单情况下的无效计算和上下文丢失。为支持该框架,构建高质量 GUI 定位数据集,并使用组相对策略优化(GRPO)训练模型,使其能同时预测点击坐标与元素边界框。在多个公开基准上的实验表明,本方法在参数量相当甚至更大的模型中达到最优性能,验证了其在高分辨率界面理解与实际 GUI 代理部署中的有效性。
原文摘要 · Abstract (English)
GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。