arXiv:2602.09856cs.CVcs.AI2026-02被引 18

用可渲染代码生成模拟界面变化,让智能体提前预判操作结果。

Code2World: A GUI World Model via Renderable Code Generation

  • 通过生成可渲染的代码预测下一界面状态,兼顾视觉质量和结构控制。
  • 在80K高质量界面-动作对上训练,实现媲美GPT-5的界面预测能力。
  • 适用于需要精准界面交互的自动化任务,如手机应用导航。

自主GUI代理通过感知界面并执行动作与环境交互。作为虚拟沙箱,GUI世界模型通过条件化动作预测赋予代理类人远见。然而,现有基于文本和像素的方法难以同时实现高视觉保真度和细粒度结构可控性。为此,我们提出Code2World,一种通过可渲染代码生成模拟下一视觉状态的视觉语言编码器。为解决数据稀缺问题,我们构建AndroidCode,将GUI轨迹转化为高保真HTML,并通过视觉反馈修正机制优化合成代码,形成超过80,000对高质量屏幕-动作样本。为适配现有视觉语言模型进行代码预测,我们首先通过监督微调(SFT)完成格式布局学习,再引入渲染感知强化学习(Render-Aware RL),以渲染结果作为奖励信号,强制保证视觉语义一致性和动作连贯性。大量实验表明,Code2World-8B在下一界面预测任务中表现最优,接近GPT-5和Gemini-3-Pro-Image水平。尤其显著提升下游导航成功率,在AndroidWorld上使Gemini-2.5-Flash导航成功率提升9.5%。代码已开源:https://github.com/AMAP-ML/Code2World。

原文摘要 · Abstract (English)

Autonomous GUI agents interact with environments by perceiving interfaces and executing actions. As a virtual sandbox, the GUI World model empowers agents with human-like foresight by enabling action-conditioned prediction. However, existing text- and pixel-based approaches struggle to simultaneously achieve high visual fidelity and fine-grained structural controllability. To this end, we propose Code2World, a vision-language coder that simulates the next visual state via renderable code generation. Specifically, to address the data scarcity problem, we construct AndroidCode by translating GUI trajectories into high-fidelity HTML and refining synthesized code through a visual-feedback revision mechanism, yielding a corpus of over 80K high-quality screen-action pairs. To adapt existing VLMs into code prediction, we first perform SFT as a cold start for format layout following, then further apply Render-Aware Reinforcement Learning which uses rendered outcome as the reward signal by enforcing visual semantic fidelity and action consistency. Extensive experiments demonstrate that Code2World-8B achieves the top-performing next UI prediction, rivaling the competitive GPT-5 and Gemini-3-Pro-Image. Notably, Code2World significantly enhances downstream navigation success rates in a flexible manner, boosting Gemini-2.5-Flash by +9.5% on AndroidWorld navigation. The code is available at https://github.com/AMAP-ML/Code2World.

GUI建模代码生成强化学习界面预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。