用扩散模型实现GUI精准定位,效果媲美传统方法。
Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
- 采用混合掩码策略提升界面元素定位精度
- 在四个平台数据集上准确率最高提升20点
- 适合追求高效、可扩展的GUI自动化研究者
自回归视觉语言模型长期主导多模态理解与图形用户界面(GUI)定位任务。近期,离散扩散视觉语言模型(DVLM)在多模态推理中表现出色,具备双向注意力、并行生成和迭代优化优势,但其在GUI定位中的潜力尚未探索。本文评估离散DVLM作为自回归模型替代方案在GUI定位中的可行性。将LLaDA-V适配为单轮动作与边界框预测任务,将其建模为从多模态输入生成文本的问题。为更好捕捉边界框的层次结构,提出结合线性与确定性掩码的混合掩码策略,在步骤成功率(SSR)上相比线性掩码训练的适配版提升最多6.1点。在涵盖网页、桌面和移动端的四个数据集上,该方法始终优于线性掩码版本,并在预训练有限的情况下表现媲美自回归模型。系统消融实验表明,增加扩散步数、生成长度和块长度可提升准确率但增加延迟,准确率在一定步数后趋于平稳。通过引入多样化的GUI领域训练数据,平均延迟降低约1.3秒,各基准测试准确率平均提升20点。结果表明,离散DVLM是实现GUI定位的有前景框架,标志着向基于扩散的GUI智能体迈出重要一步。
原文摘要 · Abstract (English)
Autoregressive (AR) vision-language models (VLMs) have long dominated multimodal understanding, reasoning, and graphical user interface (GUI) grounding. Recently, discrete diffusion vision-language models (DVLMs) have shown strong performance in multimodal reasoning, offering bidirectional attention, parallel token generation, and iterative refinement. However, their potential for GUI grounding remains unexplored. In this work, we evaluate whether discrete DVLMs can serve as a viable alternative to AR models for GUI grounding. We adapt LLaDA-V for single-turn action and bounding-box prediction, framing the task as text generation from multimodal input. To better capture the hierarchical structure of bounding-box geometry, we propose a hybrid masking schedule that combines linear and deterministic masking, improving grounding accuracy by up to 6.1 points in Step Success Rate (SSR) over the GUI-adapted LLaDA-V trained with linear masking. Evaluations on four datasets spanning web, desktop, and mobile interfaces show that the adapted diffusion model with hybrid masking consistently outperforms the linear-masked variant and performs competitively with autoregressive counterparts despite limited pretraining. Systematic ablations reveal that increasing diffusion steps, generation length, and block length improves accuracy but also increases latency, with accuracy plateauing beyond a certain number of diffusion steps. Expanding the training data with diverse GUI domains further reduces latency by about 1.3 seconds and improves grounding accuracy by an average of 20 points across benchmarks. These results demonstrate that discrete DVLMs are a promising modeling framework for GUI grounding and represent an important step toward diffusion-based GUI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。