不依赖坐标回归,用布局感知匹配实现无幻觉的GUI元素定位。
Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

- 分离指令解析与定位:冻结的多模态模型生成含布局线索的视觉描述。
- 无需回归训练,在ScreenSpot-Pro上准确率提升超20%,Mind2Web成功率提高15%以上。
- 仅用文本/图标二分类标签训练,避免微调且天然抑制坐标幻觉,适合实际交互场景。
GUI代理正从依赖元数据的大语言模型转向直接处理截图的纯视觉多模态大模型(MLLM)。核心任务是将抽象用户指令转化为精确的元素坐标。该任务面临双重挑战:传统模型语义理解不足,而端到端MLLM因细粒度感知缺陷易产生坐标幻觉。本文提出一种无回归框架:冻结的MLLM负责指令解析,专用定位模型则通过布局优先候选匹配实现无回归定位,从根本上抑制幻觉并避免昂贵微调。定位模型仅使用文本/图标二分类标签训练,无需学习坐标回归。在ScreenSpot-Pro上,相比端到端系统准确率提升超20%;在Mind2Web上,任务成功率和元素选择率均提升15%以上。结果表明,解耦指令理解与布局感知定位能有效应对GUI交互的核心挑战。
原文摘要 · Abstract (English)
GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。