通过分阶段降低帧率，让视频生成更高效。
TPDiff: Temporal Pyramid Video Diffusion Model
- 分阶段逐步提升帧率，仅最后一阶段用全帧率
- 训练成本降低50%，推理速度提升1.5倍
- 适配多种扩散模型，适合高效视频生成研究
视频扩散模型的发展面临巨大计算开销。我们发现扩散反向过程具有天然的熵减少特性，而视频模态存在帧间冗余，因此在高熵阶段无需保持全帧率。基于此，提出TPDiff统一框架，将扩散过程分为多个阶段，逐步提高帧率，仅最后一阶段使用全帧率，显著提升训练与推理效率。为训练多阶段扩散模型，设计阶段式扩散训练框架：通过解对齐数据与噪声下的分段概率流常微分方程（ODE），该策略适用于多种扩散形式，并进一步提升训练效率。全面实验验证方法通用性，实现训练成本降低50%，推理效率提升1.5倍。
原文摘要 · Abstract (English)
The development of video diffusion models unveils a significant challenge: the substantial computational demands. To mitigate this challenge, we note that the reverse process of diffusion exhibits an inherent entropy-reducing nature. Given the inter-frame redundancy in video modality, maintaining full frame rates in high-entropy stages is unnecessary. Based on this insight, we propose TPDiff, a unified framework to enhance training and inference efficiency. By dividing diffusion into several stages, our framework progressively increases frame rate along the diffusion process with only the last stage operating on full frame rate, thereby optimizing computational efficiency. To train the multi-stage diffusion model, we introduce a dedicated training framework: stage-wise diffusion. By solving the partitioned probability flow ordinary differential equations (ODE) of diffusion under aligned data and noise, our training strategy is applicable to various diffusion forms and further enhances training efficiency. Comprehensive experimental evaluations validate the generality of our method, demonstrating 50% reduction in training cost and 1.5x improvement in inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。