CascadeV用级联架构生成2K高清视频，提升分辨率与帧率。
CascadeV: An Implementation of Wurstchen Architecture for Video Generation
- 采用级联潜空间扩散模型，分步生成视频提升效率。
- 实现2K分辨率视频生成，压缩比更高，计算成本更低。
- 支持与现有模型级联，可无调优提升4倍分辨率或帧率。
近期，扩散模型在文本到图像（T2I）生成中取得巨大成功，其在文本到视频（T2V）应用中的潜力也日益受到关注。然而，扩散模型的计算需求对高分辨率、高帧率视频生成构成重大挑战。本文提出CascadeV，一种级联潜空间扩散模型（LDM），可生成当前最先进的2K分辨率视频。实验表明，该级联模型实现了更高的压缩比，显著降低了高质量视频生成的计算负担。我们还实现了一种时空交替网格3D注意力机制，有效融合时空信息，确保生成视频帧间的一致性。此外，该模型可与现有T2V模型级联，理论上可实现无需微调的4×分辨率或帧率提升。代码已开源：https://github.com/bytedance/CascadeV。
原文摘要 · Abstract (English)
Recently, with the tremendous success of diffusion models in the field of text-to-image (T2I) generation, increasing attention has been directed toward their potential in text-to-video (T2V) applications. However, the computational demands of diffusion models pose significant challenges, particularly in generating high-resolution videos with high frame rates. In this paper, we propose CascadeV, a cascaded latent diffusion model (LDM), that is capable of producing state-of-the-art 2K resolution videos. Experiments demonstrate that our cascaded model achieves a higher compression ratio, substantially reducing the computational challenges associated with high-quality video generation. We also implement a spatiotemporal alternating grid 3D attention mechanism, which effectively integrates spatial and temporal information, ensuring superior consistency across the generated video frames. Furthermore, our model can be cascaded with existing T2V models, theoretically enabling a 4$\times$ increase in resolution or frames per second without any fine-tuning. Our code is available at https://github.com/bytedance/CascadeV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。