通过挖掘视频生成中注意力的稀疏性，实现高效推理。
Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
- 基于动态稀疏模式分类，区分空间与时间注意力头。
- 在CogVideoX和HunyuanVideo上分别提速2.28x和2.33x。
- 无需训练，适配现有扩散模型，适合部署优化场景。
扩散变压器（DiTs）主导视频生成，但其高计算开销严重限制实际应用，即使在高性能GPU上生成几秒视频仍需数十分钟。这主要源于3D全注意力对上下文长度呈二次复杂度。本文提出无需训练的Sparse VideoGen（SVG）框架，利用3D全注意力中的固有稀疏性提升推理效率。我们发现注意力头可动态分为两类：空间头（仅帧内空间相关标记主导）与时间头（仅跨帧时间相关标记主导）。基于此，SVG设计在线分析策略捕捉动态稀疏模式并预测头类型。结合新型硬件友好张量布局变换与定制核实现，SVG在CogVideoX-v1.5和HunyuanVideo上分别实现最高2.28x和2.33x端到端加速，同时保持生成质量。代码已开源。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality. Our code is open-sourced and is available at https://github.com/svg-project/Sparse-VideoGen
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。