arXiv:2604.02345cs.LGcs.AI2026-04被引 4

用环境反馈训练界面代理,突破人工数据瓶颈,实现高效自动化。

UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics

论文配图:UI-Oceanus: Scaling GUI Agents with Synthetic Environmental Dynamics
图 1 · 摘自论文原文
  • 通过预测界面未来状态来学习交互规律,替代模仿人类操作。
  • 合成数据量越大,导航成功率越高,线上任务提升16.8%。
  • 适合需要跨场景泛化和低成本部署的GUI自动化研究者。

通用图形界面代理的扩展受限于昂贵的人类示范数据和合成教师监督的“蒸馏上限”。为突破这些限制,我们提出UI-Oceanus框架,将学习重点从模仿高层轨迹转向通过真实环境反馈掌握交互物理规律。通过对自监督目标的系统研究,发现前向动力学(即未来界面状态的生成预测)是可扩展性的核心驱动力,显著优于逆向推断。UI-Oceanus利用这一洞察,将低成本自主探索转化为高密度生成式监督,构建稳健的内部世界模型。在多个模型上的实验表明:基于合成动态进行持续预训练(CPT)的模型,在离线基准上平均成功率达7%提升,真实在线导航中更实现16.8%的性能增益。此外,导航性能随合成数据量增加而提升。结果证实,以预测性建模为基础的训练路径,为具备强跨域适应性和组合泛化能力的可扩展GUI自动化提供了更优方案。

原文摘要 · Abstract (English)

Scaling generalist GUI agents is hindered by the data scalability bottleneck of expensive human demonstrations and the "distillation ceiling" of synthetic teacher supervision. To transcend these limitations, we propose UI-Oceanus, a framework that shifts the learning focus from mimicking high-level trajectories to mastering interaction physics via ground-truth environmental feedback. Through a systematic investigation of self-supervised objectives, we identify that forward dynamics, defined as the generative prediction of future interface states, acts as the primary driver for scalability and significantly outweighs inverse inference. UI-Oceanus leverages this insight by converting low-cost autonomous exploration, which is verified directly by system execution, into high-density generative supervision to construct a robust internal world model. Experimental evaluations across a series of models demonstrate the decisive superiority of our approach: models utilizing Continual Pre-Training (CPT) on synthetic dynamics outperform non-CPT baselines with an average success rate improvement of 7% on offline benchmarks, which amplifies to a 16.8% gain in real-world online navigation. Furthermore, we observe that navigation performance scales with synthetic data volume. These results confirm that grounding agents in forward predictive modeling offers a superior pathway to scalable GUI automation with robust cross-domain adaptability and compositional generalization.

GUI自动化自监督学习世界模型合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。