用代码增量生成界面,让手机智能体更真实地模拟操作。
AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents

- 基于动作-过渡约束,用可执行代码预测下个界面。
- 在CMGUIBench-500上结构重建和元素还原效果最佳。
- 适合做手机自动化测试与策略训练的高效仿真环境。
移动GUI智能体通过像素感知和触控操作来使用应用,是收集和优化长程移动交互策略的有前景接口。然而,敏感应用和隐私关键操作的真实轨迹难以获取。同时,现有仿真环境扩展成本高,GUI世界模型仍存在生成不稳定、模态覆盖有限、动作-过渡逻辑不一致等问题。为此,我们提出AppDeltaWorld,一种基于过渡约束的代码增量世界模型,将下一界面预测为可达的代码更新,而非无约束的图像或文本描述。AppDeltaWorld在动作-过渡约束下检索特定应用的层级1 HTML参考,根据当前屏幕、操作、预测的下一屏文本和检索到的结构生成层级2可执行HTML,并在浏览器渲染前插入生成的视觉资源。作为世界模型,AppDeltaWorld在Code2World评估中于CMGUIBench-500上达到最高保真度,结构布局和UI元素重建显著优于仅图像或仅代码的基线。作为训练环境,其支持过滤后的闭环SFT数据构建,结合公开监督信号,使AppDeltaAgent在AndroidLens上实现最先进性能,在MobileGym和MobileWorld上也保持一致提升。此外,基于世界模型的测试时强化学习可实现策略自适应,且无需与真实应用额外交互即可进一步优化。
原文摘要 · Abstract (English)
Mobile GUI agents can operate apps through pixel perception and touch actions, making them a promising interface for collecting and improving long-horizon mobile interaction policies. However, real trajectories are difficult to obtain for sensitive apps and privacy-critical operations. At the same time, existing simulated environments are costly to scale up, and GUI world models still suffer from unstable generation, limited modality coverage, and inconsistent action-transition logic. To address these limitations, we propose AppDeltaWorld, a transition-grounded delta code world model that predicts the next GUI as a reachable code update rather than as an unconstrained image or text description. AppDeltaWorld retrieves app-specific Level-1 HTML references under an action-transition constraint, generates Level-2 executable HTML conditioned on the current screen, action, predicted next-screen text, and retrieved structure, and inserts generated visual assets into image slots before browser rendering. As a world model, AppDeltaWorld achieves the highest fidelity on CMGUIBench-500 under Code2World evaluation, with clear gains in structural layout and UI element reconstruction over image-only and code-only baselines. As a training environment, AppDeltaWorld supports filtered closed-loop SFT data construction that, when combined with public supervision, enables AppDeltaAgent to achieve state-of-the-art performance on AndroidLens and consistent gains on MobileGym and MobileWorld. Moreover, world-model-based test-time reinforcement learning enables policy adaptation and shows further improvements without additional interaction with real apps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。