用压缩潜空间加速视频生成，140亿参数模型快一个数量级。
FSVideo: Fast Speed Video Diffusion Model in a Highly-Compressed Latent Space
- 采用64×64×4压缩率的视频自编码器，实现高保真重建。
- 140亿参数扩散变换器加多分辨率生成，速度比同类模型快10倍。
- 适合需要快速生成高质量视频的场景，如实时创作与交互应用。
我们提出FSVideo，一种基于Transformer的图像到视频扩散框架，旨在实现高速视频生成。核心组件包括：1）新型视频自编码器，将时空维度压缩至64×64×4，保持良好重建质量；2）改进的扩散变换器（DIT），通过新设计的层内存机制增强跨层信息流动与上下文复用；3）基于少数步骤的多分辨率生成策略，配合一个140亿参数的DIT上采样器提升视频细节。最终模型在性能上媲美主流开源模型，但推理速度提升一个数量级。本文还详细讨论了模型设计与训练策略。
原文摘要 · Abstract (English)
We introduce FSVideo, a fast speed transformer-based image-to-video (I2V) diffusion framework. We build our framework on the following key components: 1.) a new video autoencoder with highly-compressed latent space ($64\times64\times4$ spatial-temporal downsampling ratio), achieving competitive reconstruction quality; 2.) a diffusion transformer (DIT) architecture with a new layer memory design to enhance inter-layer information flow and context reuse within DIT, and 3.) a multi-resolution generation strategy via a few-step DIT upsampler to increase video fidelity. Our final model, which contains a 14B DIT base model and a 14B DIT upsampler, achieves competitive performance against other popular open-source models, while being an order of magnitude faster. We discuss our model design as well as training strategies in this report.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。