arXiv:2506.08012cs.AIcs.CV2025-06NeurIPS被引 19

让GUI模型学会自我反思与纠错,提升自动化鲁棒性。

GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

  • 通过自动构建反射数据,训练模型具备自我反思能力。
  • 在移动端实现在线迭代调优,持续提升纠错性能。
  • 无需人工标注,适合智能自动化与人机交互研究者。

多模态大语言模型在图形用户界面自动化中展现出巨大潜力,但现有模型主要依赖近乎无错的离线轨迹学习,缺乏自我反思与错误恢复能力。为此,我们提出GUI-Reflection框架,在端到端多模态GUI模型中显式集成自我反思与错误纠正能力,涵盖专用预训练、离线监督微调和在线反思调优三个阶段。该框架通过自动化数据生成流程,从已有成功轨迹中构建反射与纠错数据;提出GUI-Reflection任务集,专门用于学习与评估反思能力;构建多样化高效移动端训练环境,并设计迭代式在线反思调优算法,使模型可持续优化自身能力。实验表明,该框架显著提升了模型在复杂场景下的鲁棒性与适应性,所有数据、模型、环境与工具将公开释放。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have shown great potential in revolutionizing Graphical User Interface (GUI) automation. However, existing GUI models mostly rely on learning from nearly error-free offline trajectories, thus lacking reflection and error recovery capabilities. To bridge this gap, we propose GUI-Reflection, a novel framework that explicitly integrates self-reflection and error correction capabilities into end-to-end multimodal GUI models throughout dedicated training stages: GUI-specific pre-training, offline supervised fine-tuning (SFT), and online reflection tuning. GUI-reflection enables self-reflection behavior emergence with fully automated data generation and learning processes without requiring any human annotation. Specifically, 1) we first propose scalable data pipelines to automatically construct reflection and error correction data from existing successful trajectories. While existing GUI models mainly focus on grounding and UI understanding ability, we propose the GUI-Reflection Task Suite to learn and evaluate reflection-oriented abilities explicitly. 2) Furthermore, we built a diverse and efficient environment for online training and data collection of GUI models on mobile devices. 3) We also present an iterative online reflection tuning algorithm leveraging the proposed environment, enabling the model to continuously enhance its reflection and error correction abilities. Our framework equips GUI agents with self-reflection and correction capabilities, paving the way for more robust, adaptable, and intelligent GUI automation, with all data, models, environments, and tools to be released publicly.

GUI自动化自反思多模态模型智能代理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。