用逆向任务生成法自动构建高质量GUI操作数据,提升智能体自动化能力。
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

- 先让智能体交互环境,再反推生成任务,打破传统预设任务限制
- 通过轨迹奖励模型确保数据质量,合成数据在多个基准上显著领先
- 适合研究GUI自动化、视觉语言模型应用的开发者和研究员
基于视觉-语言模型的图形用户界面(GUI)智能体已展现出类人计算机控制能力。然而,高质量轨迹数据的获取仍是核心瓶颈:现有方法依赖人工标注或预定义任务生成合成数据,成本高且数据质量难保障,同时存在多样性不足与真实场景差距大的问题。为此,我们提出OS-Genesis,一种逆转传统轨迹采集流程的新颖数据合成管道。该方法允许智能体先感知环境并执行逐步交互,再回溯推导出高质量任务,实现轨迹级探索。通过轨迹奖励模型保证生成轨迹的质量。实验表明,使用OS-Genesis训练的GUI智能体在高难度在线基准测试中性能显著提升。深入分析验证了其高效性、更优的数据质量和多样性,优于现有合成方法。代码、数据与模型检查点已公开于https://qiushisun.github.io/OS-Genesis-Home/。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. Despite their utility in advancing digital automation, a critical bottleneck persists: collecting high-quality trajectory data for training. Common practices for collecting such data rely on human supervision or synthetic data generation through executing pre-defined tasks, which are either resource-intensive or unable to guarantee data quality. Moreover, these methods suffer from limited data diversity and significant gaps between synthetic data and real-world environments. To address these challenges, we propose OS-Genesis, a novel GUI data synthesis pipeline that reverses the conventional trajectory collection process. Instead of relying on pre-defined tasks, OS-Genesis enables agents first to perceive environments and perform step-wise interactions, then retrospectively derive high-quality tasks to enable trajectory-level exploration. A trajectory reward model is then employed to ensure the quality of the generated trajectories. We demonstrate that training GUI agents with OS-Genesis significantly improves their performance on highly challenging online benchmarks. In-depth analysis further validates OS-Genesis's efficiency and its superior data quality and diversity compared to existing synthesis methods. Our codes, data, and checkpoints are available at https://qiushisun.github.io/OS-Genesis-Home/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。