通过分块并行生成,实现超长视频的高效低耗生成。
Minute-Long Videos with Dual Parallelisms
- 将视频帧与模型层同时跨多卡并行,打破传统延迟瓶颈。
- 生成1025帧视频时,延迟降低6.54倍,内存减少1.48倍。
- 适合需要生成超长视频的AI艺术、影视创作场景。
基于扩散变换器(DiT)的视频扩散模型虽能大规模生成高质量视频,但对长视频存在处理延迟高、内存消耗大的问题。为此,本文提出一种新型分布式推理策略DualParal。核心思想是将视频帧与模型层同时在多张GPU间并行处理。然而,由于扩散模型要求各帧噪声水平同步,简单分割会导致并行性被串行化。为此,我们采用分块去噪机制:按帧块顺序通过流水线,逐步降低噪声水平。每张GPU处理特定帧块和层子集,并将前序结果传递给下一张,实现异步计算与通信。为进一步优化性能,引入两项增强:一是在每张卡上建立特征缓存,复用前一块特征作为上下文,减少跨卡通信与重复计算;二是采用协同噪声初始化策略,在不增加资源成本的前提下,共享初始噪声模式,保证全局时间动态一致性。该方法可实现快速、无伪影且无限长度的视频生成。应用于最新扩散变换器视频生成器,在8×RTX 4090上高效生成1025帧视频,最高实现6.54倍延迟降低与1.48倍内存节省。
原文摘要 · Abstract (English)
Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos. To address this, we propose a novel distributed inference strategy, termed DualParal. The core idea is that, instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs. However, a naive implementation of this division faces a key limitation: since diffusion models require synchronized noise levels across frames, this implementation leads to the serialization of original parallelisms. We leverage a block-wise denoising scheme to handle this. Namely, we process a sequence of frame blocks through the pipeline with progressively decreasing noise levels. Each GPU handles a specific block and layer subset while passing previous results to the next GPU, enabling asynchronous computation and communication. To further optimize performance, we incorporate two key enhancements. Firstly, a feature cache is implemented on each GPU to store and reuse features from the prior block as context, minimizing inter-GPU communication and redundant computation. Secondly, we employ a coordinated noise initialization strategy, ensuring globally consistent temporal dynamics by sharing initial noise patterns across GPUs without extra resource costs. Together, these enable fast, artifact-free, and infinitely long video generation. Applied to the latest diffusion transformer video generator, our method efficiently produces 1,025-frame videos with up to 6.54$\times$ lower latency and 1.48$\times$ lower memory cost on 8$\times$RTX 4090 GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。