无需训练，动态搜索界面元素，提升大模型理解复杂界面的能力
DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding
- 通过模拟人眼聚焦、移动、扫视三种视觉动作，逐步定位目标区域
- 在ScreenSpot-Pro上使通用与专用模型准确率提升14%
- 可无缝接入现有多模态大模型，适合开发GUI智能助手的工程师
由多模态大语言模型（MLLMs）驱动的GUI代理在理解与执行用户指令方面展现出强大能力。然而，从高分辨率截图中精准定位与指令相关的元素仍具挑战，尤其当界面包含大量无关组件时。受人类在复杂屏幕上动态调整感知范围以定位任务相关区域的启发，我们提出DRS-GUI——一种无需训练的动态区域搜索框架，可无缝集成至现有MLLMs。DRS-GUI引入轻量级UI Perceptor，执行聚焦、移位、散射三种类人感知动作，逐步探索界面并生成区域候选。为动态调度这些动作，我们设计基于蒙特卡洛树搜索（MCTS）的动作规划器，并采用区域质量奖励机制评估与选择高相关性区域，高效剔除冗余UI元素。实验表明，DRS-GUI在ScreenSpot-Pro基准上使通用及专用MLLM（Qwen2.5-VL-7B和UGround-V1-7B）准确率提升14%，显著增强接地性能与泛化能力。
原文摘要 · Abstract (English)
GUI agents powered by Multimodal Large Language Models (MLLMs) have demonstrated impressive capability in understanding and executing user instructions. However, accurately grounding instruction-relevant elements from high-resolution screenshots cluttered with irrelevant UI components remains challenging for existing approaches. Inspired by how humans dynamically adjust their perceptual scope to locate task-related regions on complex screens, we propose DRS-GUI, a training-free dynamic region search framework for GUI grounding that can be seamlessly integrated into existing MLLMs. DRS-GUI introduces a lightweight UI Perceptor that performs three human-like perceptual actions (Focus, Shift, and Scatter) to progressively explore the interface and generate region proposals. To dynamically schedule these actions, we further design an Action Planner based on Monte Carlo Tree Search (MCTS). A region quality reward is employed to evaluate and select the highly instruction-relevant region, efficiently pruning redundant UI elements. Experiments demonstrate that DRS-GUI yields a 14\% improvement on ScreenSpot-Pro for general and GUI-specific MLLMs (Qwen2.5-VL-7B and UGround-V1-7B), significantly enhancing grounding performance and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。