30B参数视频大模型,支持204帧长视频生成。
Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
- 用16×16空间+8×时间压缩的VAE提升生成效率
- 3D注意力扩散模型配合流匹配训练,生成204帧视频
- 中文英文双语提示编码,适合多语言创作者
我们提出Step-Video-T2V,一个拥有300亿参数、可生成最长204帧视频的文本到视频预训练模型。设计了深度压缩的视频变分自编码器(Video-VAE),实现16×16空间与8×时间压缩比,同时保持优异的重建质量。用户提示通过双语文本编码器处理中英文输入。采用带有3D全注意力的DiT模型,基于流匹配进行训练,将噪声逐步去噪为潜在帧。引入基于视频的直接偏好优化(Video-DPO)以减少伪影,提升视觉质量。在新构建的Step-Video-T2V-Eval基准上评估,性能超越开源及商用引擎。论文还分析当前扩散模型范式局限,并展望未来方向。代码与评测集已开源于https://github.com/stepfun-ai/Step-Video-T2V,在线版本可通过https://yuewen.cn/videos访问。目标是加速视频基础模型创新,赋能内容创作者。
原文摘要 · Abstract (English)
We present Step-Video-T2V, a state-of-the-art text-to-video pre-trained model with 30B parameters and the ability to generate videos up to 204 frames in length. A deep compression Variational Autoencoder, Video-VAE, is designed for video generation tasks, achieving 16x16 spatial and 8x temporal compression ratios, while maintaining exceptional video reconstruction quality. User prompts are encoded using two bilingual text encoders to handle both English and Chinese. A DiT with 3D full attention is trained using Flow Matching and is employed to denoise input noise into latent frames. A video-based DPO approach, Video-DPO, is applied to reduce artifacts and improve the visual quality of the generated videos. We also detail our training strategies and share key observations and insights. Step-Video-T2V's performance is evaluated on a novel video generation benchmark, Step-Video-T2V-Eval, demonstrating its state-of-the-art text-to-video quality when compared with both open-source and commercial engines. Additionally, we discuss the limitations of current diffusion-based model paradigm and outline future directions for video foundation models. We make both Step-Video-T2V and Step-Video-T2V-Eval available at https://github.com/stepfun-ai/Step-Video-T2V. The online version can be accessed from https://yuewen.cn/videos as well. Our goal is to accelerate the innovation of video foundation models and empower video content creators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。