利用帧间运动一致性复用去噪结果,加速视频生成。
Denoising Reuse: Exploiting Inter-frame Motion Consistency for Efficient Video Latent Generation
- 用轻量级运动传播复用前帧粗粒度噪声,减少重复计算。
- 在关键帧保持精细去噪,视觉质量优于传统方法。
- 适合需要高效视频生成的开发者和研究者。
基于扩散模型的视频生成因逐帧迭代去噪过程而面临高计算成本。本文提出扩散复用运动(Dr. Mo)网络以加速潜在空间视频生成。核心发现是:早期去噪步骤中的粗粒度噪声在连续视频帧间具有高度运动一致性。基于此,Dr. Mo通过设计轻量级帧间运动将这些粗粒度噪声传播至下一帧,消除帧间扩散模型中的大量计算冗余。更敏感、细粒度的噪声仍通过后续去噪步骤获取,对保留视觉质量至关重要。决定何时从运动传播切换到去噪是效率与质量的关键权衡。为此,Dr. Mo引入元网络Denoising Step Selector(DSS),动态确定各帧理想的中间步骤。在视频生成与编辑任务上的广泛评估表明,Dr. Mo能显著加速扩散模型,同时提升视觉质量。
原文摘要 · Abstract (English)
Video generation using diffusion-based models is constrained by high computational costs due to the frame-wise iterative diffusion process. This work presents a Diffusion Reuse MOtion (Dr. Mo) network to accelerate latent video generation. Our key discovery is that coarse-grained noises in earlier denoising steps have demonstrated high motion consistency across consecutive video frames. Following this observation, Dr. Mo propagates those coarse-grained noises onto the next frame by incorporating carefully designed, lightweight inter-frame motions, eliminating massive computational redundancy in frame-wise diffusion models. The more sensitive and fine-grained noises are still acquired via later denoising steps, which can be essential to retain visual qualities. As such, deciding which intermediate steps should switch from motion-based propagations to denoising can be a crucial problem and a key tradeoff between efficiency and quality. Dr. Mo employs a meta-network named Denoising Step Selector (DSS) to dynamically determine desirable intermediate steps across video frames. Extensive evaluations on video generation and editing tasks have shown that Dr. Mo can substantially accelerate diffusion models in video tasks with improved visual qualities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。