用图文模型提升视频生成质量与速度,解决画面抖动问题。
Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis
- 将图文模型融入视频生成流程,通过去噪优化帧质量。
- 相比以往方法,视频质量显著提升,推理速度最高快4.5倍。
- 适合需要高质量视频生成且注重效率的研究者与开发者。
近年来,大规模文本到视频(T2V)生成模型因其能够根据文本描述生成视频而受到广泛关注。然而,实现高画质与有效运动表征仍是重大挑战。现有方法通常通过微调预训练的文本到图像(T2I)模型来优化视频帧,导致帧间不一致引发闪烁和伪影。本文提出EVS——一种无需训练的封装式视频合成器,通过组合T2I与T2V模型,同时提升视觉保真度与运动平滑性。该方法利用成熟的基于扩散的T2I模型,将低质量视频帧视为分布外样本,通过加噪与去噪步骤进行优化;同时借助T2V主干网络保证运动一致性。通过将T2V的时间仅先验嵌入T2I生成过程,EVS有效融合两类模型优势,生成更高质量的视频。实验验证了该方法的有效性,且推理速度相较以往提升1.6倍至4.5倍。源代码见:https://github.com/Tonniia/EVS。
原文摘要 · Abstract (English)
In recent years, large text-to-video (T2V) synthesis models have garnered considerable attention for their abilities to generate videos from textual descriptions. However, achieving both high imaging quality and effective motion representation remains a significant challenge for these T2V models. Existing approaches often adapt pre-trained text-to-image (T2I) models to refine video frames, leading to issues such as flickering and artifacts due to inconsistencies across frames. In this paper, we introduce EVS, a training-free Encapsulated Video Synthesizer that composes T2I and T2V models to enhance both visual fidelity and motion smoothness of generated videos. Our approach utilizes a well-trained diffusion-based T2I model to refine low-quality video frames by treating them as out-of-distribution samples, effectively optimizing them with noising and denoising steps. Meanwhile, we employ T2V backbones to ensure consistent motion dynamics. By encapsulating the T2V temporal-only prior into the T2I generation process, EVS successfully leverages the strengths of both types of models, resulting in videos of improved imaging and motion quality. Experimental results validate the effectiveness of our approach compared to previous approaches. Our composition process also leads to a significant improvement of 1.6x-4.5x speedup in inference time. Source codes: https://github.com/Tonniia/EVS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。