用世界模型训练手机界面智能体,减少真实交互,提升效率与成功率。
WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning

- 用世界模型替代真实环境进行强化学习,实现虚拟训练
- 在安卓基准上任务成功率显著高于基线方法,轨迹更高效
- 支持并行化训练,适合需要快速迭代的移动端智能体开发
基于强化学习(RL)的图形用户界面(GUI)智能体在移动平台上展现出强大的环境学习能力。然而,传统RL通常需要大量真实环境交互,导致资源消耗高且训练不稳定,尤其在GUI场景中尤为明显。为此,我们提出WM-R1,首个利用世界模型而非真实环境训练移动端GUI智能体的强化学习框架。具体而言,世界模型在所有轨迹生成中提供状态转移,取代真实Android环境。同时,将世界模型嵌入推理过程,使智能体能在执行前预判动作后果。关键优势在于完全无需真实环境交互,支持大规模并行化和细粒度的轨迹生成,并引入多维规则奖励,联合优化任务成功率、轨迹效率与世界模型利用率。为提升训练效率,我们构建了包含2000个挑战性任务的高质量数据集。在安卓移动端基准测试中,WM-R1训练的智能体显著优于仅使用GRPO的基线和推理时仿真方法。代码已开源:https://github.com/genalyu/WM-R1。
原文摘要 · Abstract (English)
GUI agents trained with reinforcement learning (RL) have showcased strong environment learning capabilities on mobile platforms. However, RL typically demands extensive real-environment interactions, leading to high resource costs and instability, especially in GUI scenarios. To address these, we propose WM-R1, the first reinforcement learning framework that trains mobile GUI agents with world models instead of real environments. Specifically, world models serve as the source of state transitions during all rollouts, replacing the real Android environment within the training loop. WM-R1 also embeds world models directly into the thinking process, enabling agents to reason about the consequences of candidate actions before committing to the final action. Crucially, WM-R1 eliminates the need for real-environment interaction, supports massively parallelized and step-level granularized trajectory generation grounded in world models, and introduces a multi-dimensional rule-based reward that jointly optimizes task success, trajectory efficiency, and world model utilization. For efficient training, we curate a high-quality dataset of 2000 challenging tasks. Experiments on Android mobile benchmarks demonstrate that WM-R1-trained agents significantly outperform GRPO-only baselines and inference-time simulation methods. Code is available at https://github.com/genalyu/WM-R1 .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。