用代码作为世界大脑,实现持久演化与高保真视觉生成的结合。
Code World Model: Coding Agent as World Brain

- 用语言模型生成可执行代码来维护世界状态和规则
- 通过代理视频实现视觉细节与时空约束的精准匹配
- 适合研究开放世界模拟与具身智能的学者
世界模型旨在模拟复杂环境在动作和事件下的演化过程,但现有基于视频的世界模型主要从视觉观察中学习动态,而视觉仅反映结果,难以捕捉底层知识、规则与机制,导致难以维持持久后果和连贯的开放式演化。本文提出代码世界模型(Code World Model),将世界演化与视觉呈现分离:由编码代理充当世界大脑,利用语言模型的推理与编码能力,生成可执行代码以维持持久世界状态并实现规则一致的演化;为连接代码状态与视觉生成,引入一种代理表示,编码帧级时空约束,并编译为代理视频,作为视频模型的条件以渲染高保真视觉观测。我们进一步构建了从游戏和真实视频中提取对齐的代理-观测数据对的数据流水线。在配对游戏数据上微调后,MiniMax-H3 能根据编码代理生成的时空规范,在简单交互世界中保持丰富的视觉细节与动态一致性。结果表明,结合代码实现持久演化、视频模型实现灵活视觉生成,为开放世界模型提供了新路径。
原文摘要 · Abstract (English)
World models aim to simulate how complex environments evolve under actions and events, yet existing video-based world models primarily learn dynamics from visual observations, which reveal outcomes rather than the underlying knowledge, rules, and mechanisms governing world evolution. This makes it difficult to maintain persistent consequences and support coherent, open-ended evolution. We introduce Code World Model, a framework that separates world evolution from visual realization by combining the reasoning and coding capabilities of language models with the generative priors of video models. A coding agent serves as the world brain, reasoning about events and their consequences and generating executable code to maintain persistent world state and perform rule-consistent evolution. To connect executable state with visual generation, we introduce a proxy representation that encodes frame-wise spatiotemporal constraints and is compiled into a proxy video, which conditions a video model to render high-fidelity visual observations. We further develop data pipelines for constructing aligned proxy-observation pairs from gameplay and real-world videos. After fine-tuning on paired gameplay data, MiniMax-H3 follows proxy-based spatiotemporal specifications from simple interactive worlds built by the coding agent while preserving rich visual details and dynamics. These results demonstrate the potential of combining code for persistent world evolution with video models for flexible visual realization, providing a new path toward open-ended world models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。