将慢速双向视频扩散模型改造为快速自回归生成，实现高画质流式输出。
From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
- 用自回归结构替代双向注意力，逐帧生成避免等待全序列。
- 4步生成器在长视频基准上得分84.27，超越所有已有模型。
- 支持零样本视频转视频、图生视频与动态提示，适合实时交互应用。
当前视频扩散模型虽生成质量高，但因依赖双向注意力，在交互场景中响应慢——单帧生成需处理整个序列（含未来帧）。本文将预训练的双向扩散变压器改造为自回归结构，实现逐帧即时生成。为进一步降低延迟，提出视频版分布匹配蒸馏（DMD），将50步扩散模型压缩至4步生成器。为保证蒸馏稳定性，引入基于教师ODE轨迹的学生初始化方案，以及不对称蒸馏策略：用双向教师监督因果学生模型。该方法有效缓解自回归生成中的误差累积，使仅在短片段上训练的模型也能生成长时视频。模型在VBench-Long基准上总分达84.27，超越所有现有模型；单卡实现9.4 FPS的高速流式生成（得益于KV缓存）。该方法还支持零样本视频到视频转换、图像到视频生成及动态提示。
原文摘要 · Abstract (English)
Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models. It enables fast streaming generation of high-quality videos at 9.4 FPS on a single GPU thanks to KV caching. Our approach also enables streaming video-to-video translation, image-to-video, and dynamic prompting in a zero-shot manner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。