不训练也能提升视频生成一致性，用教师模型引导采样过程
VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
- 用预训练视频模型在采样早期引导去噪过程
- 显著提升时间连贯性与图像质量，无需额外训练
- 适合希望快速优化现有视频生成模型的开发者
文本到图像（T2I）扩散模型已革新视觉内容创作，但扩展至文本到视频（T2V）生成仍面临时间一致性难题。现有方法常导致图像质量下降或计算成本过高。为此，我们提出VideoGuide框架，无需额外训练即可增强预训练T2V模型的时间一致性。该方法在推理初期利用任意预训练视频扩散模型（VDM）作为引导，将引导模型的去噪样本插值融入采样模型的去噪流程。该方法显著提升时间一致性和图像保真度，提供低成本且实用的解决方案，实现不同视频扩散模型的优势协同。此外，我们验证了先验蒸馏现象：基础模型可通过该方法借助引导模型的更优数据先验，实现更强的文本一致性。
原文摘要 · Abstract (English)
Text-to-image (T2I) diffusion models have revolutionized visual content creation, but extending these capabilities to text-to-video (T2V) generation remains a challenge, particularly in preserving temporal consistency. Existing methods that aim to improve consistency often cause trade-offs such as reduced imaging quality and impractical computational time. To address these issues we introduce VideoGuide, a novel framework that enhances the temporal consistency of pretrained T2V models without the need for additional training or fine-tuning. Instead, VideoGuide leverages any pretrained video diffusion model (VDM) or itself as a guide during the early stages of inference, improving temporal quality by interpolating the guiding model's denoised samples into the sampling model's denoising process. The proposed method brings about significant improvement in temporal consistency and image fidelity, providing a cost-effective and practical solution that synergizes the strengths of various video diffusion models. Furthermore, we demonstrate prior distillation, revealing that base models can achieve enhanced text coherence by utilizing the superior data prior of the guiding model through the proposed method. Project Page: https://dohunlee1.github.io/videoguide.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。