arXiv:2509.15738cs.LG2025-09被引 10

用随机探索+目标推理生成高质量GUI操作数据,解决自动化工具训练数据少的问题。

GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoning

  • 通过随机试错与目标引导分阶段生成真实多样的界面操作轨迹。
  • 在多个基准上实现更高轨迹熵和更丰富的交互流程覆盖。
  • 适合想提升GUI智能体泛化能力的研究者和开发者。

图形用户界面(GUI)智能体依赖大语言模型与视觉语言模型,有望实现数字环境中的端到端自动化。然而其发展受限于可扩展、高质量轨迹数据的匮乏。现有数据收集方法要么依赖昂贵且不一致的人工标注,要么采用合成生成,但在多样性与任务覆盖之间存在权衡。为此,我们提出GUI-ReWalk:一种增强推理能力的多阶段框架,用于合成真实且多样化的GUI轨迹。该框架首先通过随机探索模拟人类试错行为,再逐步过渡到由推断目标驱动的推理引导阶段,实现有目的的连贯交互。同时支持多步任务生成,可在多个应用间构建长程工作流。通过结合随机性以保证多样性、目标感知推理以确保结构合理性,GUI-ReWalk生成的数据更贴近人类与计算机交互中意图驱动、自适应的特征。我们基于GUI-ReWalk数据集训练Qwen2.5-VL-7B,并在Screenspot-Pro、OSWorld-G、UI-Vision、AndroidControl和GUI-Odyssey等多个基准上评估。结果表明,GUI-ReWalk显著提升了交互流的覆盖率、轨迹熵,并更真实地反映用户意图。这证明其为推进GUI智能体研究提供了一种可扩展、高效的数据生成方案,助力现实世界自动化落地。

原文摘要 · Abstract (English)

Graphical User Interface (GUI) Agents, powered by large language and vision-language models, hold promise for enabling end-to-end automation in digital environments. However, their progress is fundamentally constrained by the scarcity of scalable, high-quality trajectory data. Existing data collection strategies either rely on costly and inconsistent manual annotations or on synthetic generation methods that trade off between diversity and meaningful task coverage. To bridge this gap, we present GUI-ReWalk: a reasoning-enhanced, multi-stage framework for synthesizing realistic and diverse GUI trajectories. GUI-ReWalk begins with a stochastic exploration phase that emulates human trial-and-error behaviors, and progressively transitions into a reasoning-guided phase where inferred goals drive coherent and purposeful interactions. Moreover, it supports multi-stride task generation, enabling the construction of long-horizon workflows across multiple applications. By combining randomness for diversity with goal-aware reasoning for structure, GUI-ReWalk produces data that better reflects the intent-aware, adaptive nature of human-computer interaction. We further train Qwen2.5-VL-7B on the GUI-ReWalk dataset and evaluate it across multiple benchmarks, including Screenspot-Pro, OSWorld-G, UI-Vision, AndroidControl, and GUI-Odyssey. Results demonstrate that GUI-ReWalk enables superior coverage of diverse interaction flows, higher trajectory entropy, and more realistic user intent. These findings establish GUI-ReWalk as a scalable and data-efficient framework for advancing GUI agent research and enabling robust real-world automation.

GUI智能体数据生成多模态自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。