通过分步对角蒸馏,实现高效实时视频生成。
Streaming Autoregressive Video Generation via Diagonal Distillation
- 采用非对称生成策略:前期多步、后期少步,增强时序信息利用。
- 5秒视频仅需2.61秒生成,速度提升277.3倍,保持高运动一致性。
- 适合需要低延迟、高质量视频生成的实时应用开发者。
大型预训练扩散模型显著提升了生成视频的质量,但其在实时流式传输中的应用仍受限。自回归模型天然适用于逐帧生成,但高保真度需大量计算。扩散蒸馏可将模型压缩为少步高效版本,但现有视频蒸馏方法多沿用图像专用技术,忽视时序依赖,导致运动连贯性差、误差累积严重、延迟与质量难以兼顾。我们识别出两大原因:步数减少时对时序上下文利用不足,以及下一区块预测中隐含的噪声水平预测偏差(暴露偏差)。为此提出对角蒸馏(Diagonal Distillation),独立于现有方法,更充分挖掘视频块间与去噪步骤间的时序信息。核心是不对称生成策略:早期多步处理,后期少步推断。使后段块继承前段充分去噪后的外观信息,并以部分去噪块为条件输入进行后续合成。通过对齐块生成中隐含的噪声预测与实际推理条件,有效缓解误差传播,减少长序列过饱和现象。进一步引入隐式光流建模,在严格步数约束下保持运动质量。本方法在2.61秒内生成5秒视频(最高达31 FPS),相比未蒸馏模型提速277.3倍。
原文摘要 · Abstract (English)
Large pretrained diffusion models have significantly enhanced the quality of generated videos, and yet their use in real-time streaming remains limited. Autoregressive models offer a natural framework for sequential frame synthesis but require heavy computation to achieve high fidelity. Diffusion distillation can compress these models into efficient few-step variants, but existing video distillation approaches largely adapt image-specific methods that neglect temporal dependencies. These techniques often excel in image generation but underperform in video synthesis, exhibiting reduced motion coherence, error accumulation over long sequences, and a latency-quality trade-off. We identify two factors that result in these limitations: insufficient utilization of temporal context during step reduction and implicit prediction of subsequent noise levels in next-chunk prediction (i.e., exposure bias). To address these issues, we propose Diagonal Distillation, which operates orthogonally to existing approaches and better exploits temporal information across both video chunks and denoising steps. Central to our approach is an asymmetric generation strategy: more steps early, fewer steps later. This design allows later chunks to inherit rich appearance information from thoroughly processed early chunks, while using partially denoised chunks as conditional inputs for subsequent synthesis. By aligning the implicit prediction of subsequent noise levels during chunk generation with the actual inference conditions, our approach mitigates error propagation and reduces oversaturation in long-range sequences. We further incorporate implicit optical flow modeling to preserve motion quality under strict step constraints. Our method generates a 5-second video in 2.61 seconds (up to 31 FPS), achieving a 277.3x speedup over the undistilled model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。