用双向扩散采样提升视频插帧质量,无需多次重去噪。
ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler
- 顺序采样前后路径,分别以首尾帧为条件生成中间帧。
- 1024×576分辨率下25帧插值仅需195秒,效率领先。
- 适合需要高质量视频插帧的科研与影视制作场景。
近期大规模文本到视频(T2V)与图像到视频(I2V)扩散模型在视频生成方面取得显著进展,尤其在关键帧插值方面。然而,现有图像到视频扩散模型虽能从单张条件帧生成视频,但对双帧(起始帧与结束帧)条件生成仍需适配,这对有效有界插值至关重要。当前融合前后路径并行的方法常因离域问题导致伪影,或需多次迭代重去噪。本文提出一种新型双向采样策略,通过依次沿前后路径采样,分别以起始帧和结束帧为条件,确保中间帧生成更连贯且位于流形上。同时引入先进引导技术CFG++与DDS进一步优化插值过程。集成后方法达到当前最优性能,在单张3090显卡上可于195秒内完成1024×576分辨率下25帧的插值,成为关键帧插值的领先方案。
原文摘要 · Abstract (English)
Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models has greatly enhanced video generation, especially in terms of keyframe interpolation. However, current image-to-video diffusion models, while powerful in generating videos from a single conditioning frame, need adaptation for two-frame (start & end) conditioned generation, which is essential for effective bounded interpolation. Unfortunately, existing approaches that fuse temporally forward and backward paths in parallel often suffer from off-manifold issues, leading to artifacts or requiring multiple iterative re-noising steps. In this work, we introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames. Additionally, we incorporate advanced guidance techniques, CFG++ and DDS, to further enhance the interpolation process. By integrating these, our method achieves state-of-the-art performance, efficiently generating high-quality, smooth videos between keyframes. On a single 3090 GPU, our method can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds, establishing it as a leading solution for keyframe interpolation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。