用可执行代码生成界面,兼顾文字精准与视觉质量。
Generative Visual Code Mobile World Models
- 用视觉语言模型直接输出可渲染的网页代码,替代像素生成
- 在多个基准上超越50倍更大模型,实现更小体积更高精度
- 适合需要高效、高保真移动界面建模的研究者与开发者
移动图形用户界面(GUI)世界模型(WMs)为提升移动端代理在训练和推理阶段的表现提供了新路径。然而现有方法面临关键权衡:基于文本的模型牺牲视觉保真度,而视觉模型因难以精确渲染文字,依赖复杂且缓慢的外部模型流水线。本文提出一种新范式:通过可渲染代码生成视觉世界模型,即单个视觉语言模型(VLM)将下一界面状态预测为可执行网页代码,该代码可直接渲染为像素,而非直接生成像素。此方法结合了两类优势:VLM保留语言先验以精准呈现文字,同时其在结构化网页代码上的预训练支持高质量视觉生成。我们提出了gWorld(8B, 32B),首个基于该范式的开源权重移动GUI世界模型,并构建了自动合成代码训练数据的数据生成框架。在4个分布内及2个分布外基准上的广泛评估显示,gWorld在准确率与模型规模之间建立了新的帕累托前沿,优于50.25倍更大的50个前沿开源模型。进一步分析表明:(1) 通过gWorld扩展训练数据带来显著提升;(2) 我们流水线各组件均有效提高数据质量;(3) 更强的世界建模能力直接提升下游移动界面策略性能。
原文摘要 · Abstract (English)
Mobile Graphical User Interface (GUI) World Models (WMs) offer a promising path for improving mobile GUI agent performance at train- and inference-time. However, current approaches face a critical trade-off: text-based WMs sacrifice visual fidelity, while the inability of visual WMs in precise text rendering led to their reliance on slow, complex pipelines dependent on numerous external models. We propose a novel paradigm: visual world modeling via renderable code generation, where a single Vision-Language Model (VLM) predicts the next GUI state as executable web code that renders to pixels, rather than generating pixels directly. This combines the strengths of both approaches: VLMs retain their linguistic priors for precise text rendering while their pre-training on structured web code enables high-fidelity visual generation. We introduce gWorld (8B, 32B), the first open-weight visual mobile GUI WMs built on this paradigm, along with a data generation framework (gWorld) that automatically synthesizes code-based training data. In extensive evaluation across 4 in- and 2 out-of-distribution benchmarks, gWorld sets a new pareto frontier in accuracy versus model size, outperforming 8 frontier open-weight models over 50.25x larger. Further analyses show that (1) scaling training data via gWorld yields meaningful gains, (2) each component of our pipeline improves data quality, and (3) stronger world modeling improves downstream mobile GUI policy performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。