1分钟生成1分钟视频,且每秒生成耗时不足1秒。
Magic 1-For-1: Generating One Minute Video Clips within One Minute
- 分步生成:先图文再图视频,降低训练难度。
- 5秒视频3秒生成,1分钟视频1分钟完成,平均每秒生成耗时<1秒。
- 适合追求高效视频生成的开发者和研究者。
本文提出Magic141,一种高效视频生成模型,通过将文本到视频生成任务分解为文本到图像与图像到视频两个更易收敛的子任务,实现扩散步骤蒸馏。实验表明,在相同优化算法下,图像到视频任务比文本到视频任务更易收敛。我们从三个角度优化:1)利用多模态先验条件注入加速模型收敛;2)采用对抗性步骤蒸馏降低推理延迟;3)通过参数稀疏化优化内存消耗。在这些技术加持下,可实现5秒视频3秒生成。结合测试时滑动窗口策略,可在1分钟内生成1分钟视频,平均生成速度低于1秒/秒,显著提升视觉质量与运动动态。我们初步探索了计算成本与视频质量间的权衡,为开源研究提供基础模型。代码与权重已公开于https://github.com/DA-Group-PKU/Magic-1-For-1。
原文摘要 · Abstract (English)
In this technical report, we present Magic 1-For-1 (Magic141), an efficient video generation model with optimized memory consumption and inference latency. The key idea is simple: factorize the text-to-video generation task into two separate easier tasks for diffusion step distillation, namely text-to-image generation and image-to-video generation. We verify that with the same optimization algorithm, the image-to-video task is indeed easier to converge over the text-to-video task. We also explore a bag of optimization tricks to reduce the computational cost of training the image-to-video (I2V) models from three aspects: 1) model convergence speedup by using a multi-modal prior condition injection; 2) inference latency speed up by applying an adversarial step distillation, and 3) inference memory cost optimization with parameter sparsification. With those techniques, we are able to generate 5-second video clips within 3 seconds. By applying a test time sliding window, we are able to generate a minute-long video within one minute with significantly improved visual quality and motion dynamics, spending less than 1 second for generating 1 second video clips on average. We conduct a series of preliminary explorations to find out the optimal tradeoff between computational cost and video quality during diffusion step distillation and hope this could be a good foundation model for open-source explorations. The code and the model weights are available at https://github.com/DA-Group-PKU/Magic-1-For-1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。