让视频生成实时交互,1秒出片,支持无限长视频流
MotionStream: Real-Time Video Generation with Interactive Motion Controls
- 用因果注意力+记忆缓存,实现边生成边推理的实时视频流
- 单卡达29帧/秒,比现有方法快100倍,支持无限长度生成
- 适合需要实时操控视频的创作者、设计师或交互应用开发者
当前基于运动条件的视频生成方法存在延迟高(每视频数分钟)和非因果处理无法实时交互的问题。我们提出MotionStream,可在单张GPU上实现亚秒级延迟,最高达29 FPS的流式生成。通过将文本到视频模型增强为具备运动控制能力的双向教师模型,再以自强制与分布匹配蒸馏方式将其压缩为因果学生模型,实现在线推理。针对长时序、可能无限时长视频生成的三大挑战——训练数据有限但需外推至无限时长、误差累积导致画质下降、上下文窗口扩大导致计算成本增长——我们引入精心设计的滑动窗口因果注意力与注意力池。训练中结合自滚动与注意力池及键值缓存滚动,模拟固定上下文窗口下的推理外推,从而实现任意长度视频的恒定速度生成。模型在运动跟随与视频质量上达到当前最优,同时速度提升两个数量级,首次实现无限长度实时流式生成。用户可实时绘制轨迹、操控摄像机或迁移动作,即时查看结果,带来真正交互体验。
原文摘要 · Abstract (English)
Current motion-conditioned video generation methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present MotionStream, enabling sub-second latency with up to 29 FPS streaming generation on a single GPU. Our approach begins by augmenting a text-to-video model with motion control, which generates high-quality videos that adhere to the global text prompt and local motion guidance, but does not perform inference on the fly. As such, we distill this bidirectional teacher into a causal student through Self Forcing with Distribution Matching Distillation, enabling real-time streaming inference. Several key challenges arise when generating videos of long, potentially infinite time-horizons -- (1) bridging the domain gap from training on finite length and extrapolating to infinite horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without incurring growth in computational cost due to increasing context windows. A key to our approach is introducing carefully designed sliding-window causal attention, combined with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we properly simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Our models achieve state-of-the-art results in motion following and video quality while being two orders of magnitude faster, uniquely enabling infinite-length streaming. With MotionStream, users can paint trajectories, control cameras, or transfer motion, and see results unfold in real-time, delivering a truly interactive experience.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。