用游戏数据生成可实时操控的高保真世界视频，支持跨场景自由探索。
The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
- 结合游戏与真实影像数据，训练出可连续生成720p视频的模拟器。
- 16帧/秒实时运行，支持第一人称和第三人称视角，生成时长超一小时。
- 零样本泛化，能模拟游戏与现实均未覆盖的场景如办公室开车。
我们提出The Matrix，首个能够生成连续720p高保真真实场景视频流的基线真实世界模拟器，支持第一人称与第三人称视角的实时响应控制，实现对丰富动态环境的沉浸式探索。模型在有限监督数据（来自《极限竞速：地平线5》《赛博朋克2077》等3A游戏）基础上，结合大规模无监督真实影像（如东京街道），可连续生成跨越沙漠、草原、水域及城市景观的小时级未剪辑视频序列。系统以16 FPS运行，支持实时交互，并展现出零样本泛化能力——将虚拟游戏环境迁移至真实世界中不存在的场景，例如让宝马X3在办公室环境中行驶，该场景在游戏与真实数据中均未出现。该方法证明了3A游戏数据在构建鲁棒世界模型中的潜力，为数据稀缺场景下的仿真与应用提供了新路径。
原文摘要 · Abstract (English)
We present The Matrix, the first foundational realistic world simulator capable of generating continuous 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives, enabling immersive exploration of richly dynamic environments. Trained on limited supervised data from AAA games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains -- deserts, grasslands, water bodies, and urban landscapes -- in continuous, uncut hour-long sequences. Operating at 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting--an environment present in neither gaming data nor real-world sources. This approach showcases the potential of AAA game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。