用桥梁模型提升图像生成视频质量，解决扩散模型不匹配问题。
FrameBridge: Improving Image-to-Video Generation with Bridge Models
- 用数据到数据的桥梁模型替代噪声到数据过程，增强帧间一致性。
- 零样本下在MSR-VTT上FVD降低至95（原192），非零样本在UCF-101上降至122（原171）。
- 提出SAF微调和神经先验，适合想用预训练T2V模型或从头训练的研究者。
扩散模型在图像到视频（I2V）生成上取得显著进展，但其噪声到数据的生成过程与任务本质不匹配，可能影响合成质量。本文提出FrameBridge，通过基于数据到数据的桥接模型建模帧间生成过程，充分挖掘输入图像信息，提升生成过程与I2V任务的一致性。针对两种主流训练设置，提出两项新方法：首先，提出SNR对齐微调（SAF），首次实现将扩散模型转化为桥接模型，从而可复用预训练文本到视频（T2V）模型；其次，提出神经先验，在从零训练时进一步提升合成质量。在WebVid-2M和UCF-101上的实验表明，FrameBridge优于扩散基对手（零样本下MSR-VTT FVD 95 vs. 192，非零样本下UCF-101 FVD 122 vs. 171），且SAF与神经先验有效提升了桥接模型性能。
原文摘要 · Abstract (English)
Diffusion models have achieved remarkable progress on image-to-video (I2V) generation, while their noise-to-data generation process is inherently mismatched with this task, which may lead to suboptimal synthesis quality. In this work, we present FrameBridge. By modeling the frame-to-frames generation process with a bridge model based data-to-data generative process, we are able to fully exploit the information contained in the given image and improve the consistency between the generation process and I2V task. Moreover, we propose two novel techniques toward the two popular settings of training I2V models, respectively. Firstly, we propose SNR-Aligned Fine-tuning (SAF), making the first attempt to fine-tune a diffusion model to a bridge model and, therefore, allowing us to utilize the pre-trained diffusion-based text-to-video (T2V) models. Secondly, we propose neural prior, further improving the synthesis quality of FrameBridge when training from scratch. Experiments conducted on WebVid-2M and UCF-101 demonstrate the superior quality of FrameBridge in comparison with the diffusion counterpart (zero-shot FVD 95 vs. 192 on MSR-VTT and non-zero-shot FVD 122 vs. 171 on UCF-101), and the advantages of our proposed SAF and neural prior for bridge-based I2V models. The project page: https://framebridge-icml.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。