构建UI跳转图生成长序列操作轨迹，提升自动化测试覆盖率
SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis
- 分两阶段生成：先建屏幕元素跳转图，再基于图规划多样轨迹
- 生成平均14.8步的长序列，避免无效循环，提升数据多样性
- 适合做GUI自动化、移动应用测试或模型训练的数据增强
由视觉-语言模型驱动的图形用户界面（GUI）代理在自动化真实世界移动端任务方面具有潜力。然而，进展受限于缺乏从元素丰富且快速演化的应用中收集的高覆盖度、长时序交互轨迹。现有流程通常依赖昂贵的人工演示或在线策略框架，易过度采样常见路径而遗漏罕见跳转和复杂多步操作。为此，我们提出SEE，一种两阶段数据合成框架：(i) 高效探索阶段构建屏幕与元素间的显式UI跳转图；(ii) 基于图的合成阶段通过规划与受控采样组合多样化多步轨迹。该设计实现可复现、可解释的数据生成，明确防止虚假循环并支持长时序组合。在多个真实应用中，SEE生成的轨迹平均长度达14.8步，且无伪循环；基于SEE微调的代理在任务成功率和未见屏幕泛化能力上均有提升。我们将公开合成代码与数据集。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents powered by vision-language models hold promise for automating real-world mobile tasks. However, progress is limited by the lack of high-coverage, long-horizon interaction trajectories collected from element-rich and rapidly evolving apps. Existing pipelines often rely on costly human demonstrations or on-policy framework, which tends to over-sample common flows while missing rare transitions and complex multi-step procedures. To address this problem, we propose SEE, a two-stage data synthesis framework consisting of (i) an efficient exploration stage that builds an explicit UI transition graph over screens and elements, and (ii) a graph-based synthesis stage that composes diverse multi-step trajectories via planning and controlled sampling. This design yields reproducible and explainable data generation, while explicitly preventing spurious cycles and enabling long-horizon composition. Across multiple real-world apps, SEE produces trajectories with an average length of 14.8 steps while avoiding spurious loops, and agents fine-tuned on SEE achieve improved task success and generalization to unseen screens. We will publicly release our synthesis code and dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。