实时交互世界模型，25帧/秒生成分钟级高清视频
Matrix-game 2.0: An open-source real-time and streaming interactive world model
- 用自回归扩散模型实现快速生成，仅需几步推理
- 支持鼠标键盘输入，可实时响应用户动作
- 开源1200小时带交互标注数据，适合研究交互建模
最近的交互视频生成研究展示了扩散模型作为世界模型的潜力，能够捕捉复杂物理动态和交互行为。然而，现有交互世界模型依赖双向注意力和长推理步骤，严重限制了实时性能，难以模拟需要基于历史和当前动作即时更新的真实世界动态。为此，我们提出 Matrix-Game 2.0，一种通过少量步骤自回归扩散实现实时流式视频生成的交互世界模型。其框架包含三个关键组件：(1) 面向 Unreal Engine 与 GTA5 环境的可扩展数据生成管道，有效生产约 1200 小时、具有多样化交互标注的视频数据；(2) 动作注入模块，支持以帧级鼠标与键盘输入作为交互条件；(3) 基于因果架构的少步蒸馏方法，实现实时与流式视频生成。Matrix Game 2.0 可在多种场景下以 25 FPS 的超快速度生成高质量分钟级视频。我们开源模型权重与代码库，推动交互世界建模研究发展。
原文摘要 · Abstract (English)
Recent advances in interactive video generations have demonstrated diffusion model's potential as world models by capturing complex physical dynamics and interactive behaviors. However, existing interactive world models depend on bidirectional attention and lengthy inference steps, severely limiting real-time performance. Consequently, they are hard to simulate real-world dynamics, where outcomes must update instantaneously based on historical context and current actions. To address this, we present Matrix-Game 2.0, an interactive world model generates long videos on-the-fly via few-step auto-regressive diffusion. Our framework consists of three key components: (1) A scalable data production pipeline for Unreal Engine and GTA5 environments to effectively produce massive amounts (about 1200 hours) of video data with diverse interaction annotations; (2) An action injection module that enables frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step distillation based on the casual architecture for real-time and streaming video generation. Matrix Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our model weights and codebase to advance research in interactive world modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。