通过分拆图文元素,实现无需训练的GUI精准定位。
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
- 将GUI拆分为文本与图标两模态,分别推理
- 动态聚焦初始预测区域,逐级缩放细化定位
- 无需训练/标注,适合跨平台界面理解场景
在图形用户界面(GUI)中定位自然语言查询面临视觉元素多样、空间杂乱和语言模糊等挑战。本文提出DiMo-GUI,一种无需训练的GUI定位框架,核心策略为动态视觉定位与模态感知优化。该方法不将GUI视为整体图像,而是将其拆分为文本元素与图标元素,使模型能独立使用通用视觉-语言模型对各模态进行推理。当预测存在歧义或错误时,DiMo-GUI通过生成以初始预测为中心的候选关注区域,并逐步放大子区域,实现层级化精炼,有效缓解视觉密集布局带来的混淆问题。我们在标准GUI定位基准上评估该方法,结果表明其在多个基线推理流程上均取得持续提升,验证了模态分离与区域聚焦推理结合的有效性。
原文摘要 · Abstract (English)
Grounding natural language queries in graphical user interfaces (GUIs) poses unique challenges due to the diversity of visual elements, spatial clutter, and the ambiguity of language. In this paper, we introduce DiMo-GUI, a training-free framework for GUI grounding that leverages two core strategies: dynamic visual grounding and modality-aware optimization. Instead of treating the GUI as a monolithic image, our method splits the input into textual elements and iconic elements, allowing the model to reason over each modality independently using general-purpose vision-language models. When predictions are ambiguous or incorrect, DiMo-GUI dynamically focuses attention by generating candidate focal regions centered on the model's initial predictions and incrementally zooms into subregions to refine the grounding result. This hierarchical refinement process helps disambiguate visually crowded layouts without the need for additional training or annotations. We evaluate our approach on standard GUI grounding benchmarks and demonstrate consistent improvements over baseline inference pipelines, highlighting the effectiveness of combining modality separation with region-focused reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。