通过挖掘视频扩散模型中的注意力稀疏模式,实现高效加速。
Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers
- 发现三种稳定的注意力稀疏结构,可跳过3-6%的注意力头。
- 在多个视频生成模型上实现1.58~2.38倍推理加速,保持高质量输出。
- 适合追求高效视频生成的开发者和部署人员使用。
尽管扩散变压器(DiTs)在视频生成领域取得突破,但长序列生成仍受限于注意力机制的二次复杂度,导致显著推理延迟。通过对视频扩散变压器(vDiT)注意力图的详细分析,我们识别出三种重复出现的稀疏模式:对角线、多对角线和垂直条纹结构,甚至3-6%的注意力头可被跳过。关键的是,这些模式表现出强烈的层深与头位置相关性,但对输入内容依赖较弱。基于此,我们提出Sparse-vDiT,一个针对vDiT的稀疏加速框架,包含:1)针对每种稀疏模式设计的优化稀疏核,替代密集注意力以实现高效计算;2)离线稀疏扩散搜索算法,通过硬件感知成本建模为每层每头选择最优稀疏计算策略。确定最优配置后,融合同层中采用相同策略的头以提升推理效率。集成到当前最先进的vDiT模型(CogVideoX1.5、HunyuanVideo和Wan2.1)中,Sparse-vDiT实现了理论FLOP减少2.09×、2.38×和1.67×,实际推理速度提升1.76×、1.85×和1.58×,同时保持高视觉保真度,PSNR分别达到24.13、27.09和22.59。本工作表明,vDiT中的潜在结构稀疏性可系统性地用于长视频合成。
原文摘要 · Abstract (English)
While Diffusion Transformers (DiTs) have achieved breakthroughs in video generation, this long sequence generation task remains constrained by the quadratic complexity of attention mechanisms, resulting in significant inference latency. Through detailed analysis of attention maps in Video Diffusion Transformer (vDiT), we identify three recurring sparsity patterns: diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\% attention heads can be skipped. Crucially, these patterns exhibit strong layer-depth and head-position correlations but show limited dependence on the input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels that replace dense attention with computationally efficient implementations for each identified sparsity pattern. 2) An offline sparse diffusion search algorithm that selects the optimal sparse computation strategy per layer and head via hardware-aware cost modeling. After determining the optimal configuration, we fuse heads within the same layer that share the same attention strategy, enhancing inference efficiency. Integrated into state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1), Sparse-vDiT achieves 2.09$\times$, 2.38$\times$, and 1.67$\times$ theoretical FLOP reduction, and actual inference speedups of 1.76$\times$, 1.85$\times$, and 1.58$\times$, respectively, while maintaining high visual fidelity, with PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent structural sparsity in vDiTs can be systematically exploited for long video synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。