用更少参数和更快速度,实现高质量视频帧插值。
TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation
- 设计3D小波门控与时序感知自编码器,高效提取视频时序信息。
- 在挑战性数据集上FID提升20%,参数减少3倍,推理速度加快2.3倍。
- 仅需9000倍少的训练数据,适合资源受限场景下的视频生成应用。
视频帧插值(VFI)旨在根据两个相邻帧 $I_0$ 和 $I_1$ 预测中间帧 $I_n$(n 表示时间,避免与扩散模型的时间步 $t$ 冲突)。近期方法采用图像或视频级扩散模型,在该任务上表现优异。但图像级扩散模型难以捕捉时序信息,且效率低于非扩散方法;视频级扩散模型虽能提取时序特征,却存在训练规模大、模型庞大、推理慢的问题。为此,我们提出时序感知潜空间布朗桥扩散模型(TLB-VFI),一种高效的视频级扩散模型。通过提出的3D小波门控与时序感知自编码器,从视频输入中提取丰富时序信息,使在最具有挑战性的数据集上,相比最新图像级扩散模型取得20%的FID改进。同时,得益于强时序建模能力,本方法仅需3倍少的参数,即可实现2.3倍的推理加速。引入光流引导后,训练数据需求降低9000倍,参数量比现有视频级扩散模型减少20倍以上。代码与结果详见项目页:https://zonglinl.github.io/tlbvfi_page。
原文摘要 · Abstract (English)
Video Frame Interpolation (VFI) aims to predict the intermediate frame $I_n$ (we use n to denote time in videos to avoid notation overload with the timestep $t$ in diffusion models) based on two consecutive neighboring frames $I_0$ and $I_1$. Recent approaches apply diffusion models (both image-based and video-based) in this task and achieve strong performance. However, image-based diffusion models are unable to extract temporal information and are relatively inefficient compared to non-diffusion methods. Video-based diffusion models can extract temporal information, but they are too large in terms of training scale, model size, and inference time. To mitigate the above issues, we propose Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation (TLB-VFI), an efficient video-based diffusion model. By extracting rich temporal information from video inputs through our proposed 3D-wavelet gating and temporal-aware autoencoder, our method achieves 20% improvement in FID on the most challenging datasets over recent SOTA of image-based diffusion models. Meanwhile, due to the existence of rich temporal information, our method achieves strong performance while having 3times fewer parameters. Such a parameter reduction results in 2.3x speed up. By incorporating optical flow guidance, our method requires 9000x less training data and achieves over 20x fewer parameters than video-based diffusion models. Codes and results are available at our project page: https://zonglinl.github.io/tlbvfi_page.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。