arXiv:2603.25746cs.CV2026-03被引 19

实现秒级响应的交互式多镜头视频生成,支持动态剧情修改。

ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytelling

  • 采用因果架构,基于历史上下文逐帧生成新镜头。
  • 单卡16帧/秒,生成质量媲美更慢的双向模型。
  • 适合实时互动故事创作,如游戏、虚拟主播场景。

多镜头视频生成对长篇叙事至关重要,但现有双向架构存在交互性差、延迟高等问题。本文提出ShotStream,一种新型因果多镜头架构,支持交互式叙事与即时帧生成。通过将任务重构为基于历史上下文的下一镜头生成,用户可实时发送流式提示动态调整剧情。方法首先微调文本到视频模型为双向下一镜头生成器,再通过分布匹配蒸馏转化为因果学生模型。为解决镜头间一致性与误差累积问题,引入双缓存机制:全局缓存保留条件帧以维持跨镜头连贯性,局部缓存存储当前镜头生成帧以保障内部一致性,并使用RoPE不连续指示符明确区分二者。同时提出两阶段蒸馏策略:先在真实历史帧上进行镜头内自强制,再逐步扩展至自生成历史的镜头间自强制,有效弥合训练与推理差距。大量实验表明,ShotStream在单张GPU上实现16 FPS的亚秒级延迟,生成视频在连贯性与质量上达到或超过更慢的双向模型,为实时交互式叙事提供可能。训练与推理代码及模型已开源。

原文摘要 · Abstract (English)

Multi-shot video generation is crucial for long narrative storytelling, yet current bidirectional architectures suffer from limited interactivity and high latency. We propose ShotStream, a novel causal multi-shot architecture that enables interactive storytelling and efficient on-the-fly frame generation. By reformulating the task as next-shot generation conditioned on historical context, ShotStream allows users to dynamically instruct ongoing narratives via streaming prompts. We achieve this by first fine-tuning a text-to-video model into a bidirectional next-shot generator, which is then distilled into a causal student via Distribution Matching Distillation. To overcome the challenges of inter-shot consistency and error accumulation inherent in autoregressive generation, we introduce two key innovations. First, a dual-cache memory mechanism preserves visual coherence: a global context cache retains conditional frames for inter-shot consistency, while a local context cache holds generated frames within the current shot for intra-shot consistency. And a RoPE discontinuity indicator is employed to explicitly distinguish the two caches to eliminate ambiguity. Second, to mitigate error accumulation, we propose a two-stage distillation strategy. This begins with intra-shot self-forcing conditioned on ground-truth historical shots and progressively extends to inter-shot self-forcing using self-generated histories, effectively bridging the train-test gap. Extensive experiments demonstrate that ShotStream generates coherent multi-shot videos with sub-second latency, achieving 16 FPS on a single GPU. It matches or exceeds the quality of slower bidirectional models, paving the way for real-time interactive storytelling. Training and inference code, as well as the models, are available on our

视频生成交互式多镜头实时

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。