BLADE通过联合优化稀疏注意力与步骤蒸馏,实现视频生成的高效加速且不降质。
BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation
- 设计自适应块稀疏注意力,动态聚焦关键时空特征。
- 在蒸馏中直接融合稀疏性,实现14.10倍端到端加速。
- 无需高质量数据训练,适合长视频生成场景。
扩散Transformer目前在高质量视频生成领域占据主导地位,但其缓慢的迭代去噪过程以及长序列下二次方级的注意力开销造成了显著的推理瓶颈。尽管步骤蒸馏和稀疏注意力各自展现出加速潜力,但二者有效结合面临挑战:训练无关集成效果不佳,而先蒸馏后训练稀疏注意力则需昂贵的高质量视频数据。为此,我们提出BLADE,一种创新的数据无关联合训练框架,包含:(1) 自适应块稀疏注意力(ASA)机制,动态生成内容感知的稀疏掩码,聚焦于显著的时空特征;(2) 面向稀疏性的步骤蒸馏范式,基于轨迹分布匹配(TDM),将稀疏性直接融入蒸馏过程,而非作为独立压缩步骤,并具备快速收敛特性。我们在CogVideoX-5B和Wan2.1-1.3B等文本到视频模型上验证了该框架,表现出显著的效率提升。在Wan2.1-1.3B上,相比50步基线,实现了14.10倍的端到端推理加速;在短视频长度的CogVideoX-5B上,实现了8.89倍的加速。关键的是,加速同时带来质量提升:在VBench-2.0基准上,使CogVideoX-5B得分从0.534提升至0.569,Wan2.1-1.3B从0.563升至0.570,人类评估也证实了更优表现。
原文摘要 · Abstract (English)
Diffusion Transformers currently lead the field in high-quality video generation, but their slow iterative denoising process and prohibitive quadratic attention costs for long sequences create significant inference bottlenecks. While both step distillation and sparse attention mechanisms have shown promise as independent acceleration strategies, effectively combining these approaches presents critical challenges -- training-free integration yields suboptimal results, while separately training sparse attention after step distillation requires prohibitively expensive high-quality video data. To overcome these limitations, we propose BLADE, an innovative data-free joint training framework that introduces: (1) an Adaptive Block-Sparse Attention (ASA) mechanism for dynamically generating content-aware sparsity masks to focus computation on salient spatiotemporal features, and (2) a sparsity-aware step distillation paradigm, built upon Trajectory Distribution Matching (TDM), directly incorporates sparsity into the distillation process rather than treating it as a separate compression step and features fast convergence. We validate BLADE on text-to-video models like CogVideoX-5B and Wan2.1-1.3B, and our framework demonstrates remarkable efficiency gains across different scales. On Wan2.1-1.3B, BLADE achieves a 14.10x end-to-end inference acceleration over a 50-step baseline. Moreover, on models such as CogVideoX-5B with short video sequence lengths, our framework delivers a robust 8.89x speedup. Crucially, the acceleration is accompanied by a consistent quality improvement. On the VBench-2.0 benchmark, BLADE boosts the score of CogVideoX-5B to 0.569 (from 0.534) and Wan2.1-1.3B to 0.570 (from 0.563), results that are further corroborated by superior ratings in human evaluations. Project is available at http://ziplab.co/BLADE-Homepage/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。