用神经网络实现真实世界视频的实时交互生成
From Virtual Games to Real-World Play
- 基于用户控制信号逐段生成逼真视频，支持低延迟反馈
- 无需真实动作标注，训练结合游戏数据与真实视频
- 可将虚拟控制迁移到真实场景，还能操控自行车行人
我们提出RealPlay，一种基于神经网络的真实世界游戏引擎，可从用户控制信号生成交互式视频。与以往侧重游戏风格视觉不同，RealPlay致力于生成逼真且时间一致的视频序列，模拟真实影像。系统采用交互循环：用户观察生成场景，发出控制指令，随即获得一段短视频响应。为实现高真实感和快速响应，我们解决了迭代分块预测、跨轮次时序一致性及精确控制响应等关键挑战。RealPlay在标注游戏数据与未标注真实视频混合数据上训练，无需真实动作标注。显著发现两类泛化能力：(1) 控制迁移——有效将虚拟控制信号映射至真实场景；(2) 实体迁移——尽管训练标签仅来自赛车游戏，但可控制自行车、行人等非车辆实体。项目主页：https://wenqsun.github.io/RealPlay/
原文摘要 · Abstract (English)
We introduce RealPlay, a neural network-based real-world game engine that enables interactive video generation from user control signals. Unlike prior works focused on game-style visuals, RealPlay aims to produce photorealistic, temporally consistent video sequences that resemble real-world footage. It operates in an interactive loop: users observe a generated scene, issue a control command, and receive a short video chunk in response. To enable such realistic and responsive generation, we address key challenges including iterative chunk-wise prediction for low-latency feedback, temporal consistency across iterations, and accurate control response. RealPlay is trained on a combination of labeled game data and unlabeled real-world videos, without requiring real-world action annotations. Notably, we observe two forms of generalization: (1) control transfer-RealPlay effectively maps control signals from virtual to real-world scenarios; and (2) entity transfer-although training labels originate solely from a car racing game, RealPlay generalizes to control diverse real-world entities, including bicycles and pedestrians, beyond vehicles. Project page can be found: https://wenqsun.github.io/RealPlay/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。