用低分辨率注意力引导加速视频扩散模型，生成8秒720p视频提速至分钟级。
DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidance
- 通过下采样特征图生成低分辨率注意力图，指导全分辨率稀疏计算。
- 在相同质量下，端到端速度最高提升1.75倍，显著降低延迟。
- 无需训练，适配现有模型，适合需要高效视频生成的开发者。
基于扩散变换器的视频生成模型（DiTs）因生成质量优异而备受关注，但其计算成本仍是主要瓶颈——仅注意力机制就占总延迟的80%以上，生成8秒720p视频需数十分钟，严重制约实际应用与扩展性。为此，我们提出DraftAttention，一种无需训练的框架，利用GPU上的动态稀疏注意力实现视频扩散变压器的加速。我们在压缩的潜在空间中对每帧特征图进行下采样，构建覆盖数十万潜在标记的更高层次感受野。由草稿查询和键生成的低分辨率注意力图，揭示了单帧内空间冗余及跨帧时间冗余。根据该注意力图重排查询、键和值，以指导全分辨率的稀疏注意力计算，并在计算后恢复原始顺序。这种重排实现了与硬件优化执行对齐的结构化稀疏性。理论分析表明，低分辨率草稿注意力能紧密逼近完整注意力，为构建准确稀疏注意力提供可靠引导。实验结果表明，该方法在视频生成质量上优于现有稀疏注意力方案，并在GPU上实现高达1.75倍的端到端加速。代码已开源：https://github.com/shawnricecake/draft-attention
原文摘要 · Abstract (English)
Diffusion transformer-based video generation models (DiTs) have recently attracted widespread attention for their excellent generation quality. However, their computational cost remains a major bottleneck-attention alone accounts for over 80% of total latency, and generating just 8 seconds of 720p video takes tens of minutes-posing serious challenges to practical application and scalability. To address this, we propose the DraftAttention, a training-free framework for the acceleration of video diffusion transformers with dynamic sparse attention on GPUs. We apply down-sampling to each feature map across frames in the compressed latent space, enabling a higher-level receptive field over the latent composed of hundreds of thousands of tokens. The low-resolution draft attention map, derived from draft query and key, exposes redundancy both spatially within each feature map and temporally across frames. We reorder the query, key, and value based on the draft attention map to guide the sparse attention computation in full resolution, and subsequently restore their original order after the attention computation. This reordering enables structured sparsity that aligns with hardware-optimized execution. Our theoretical analysis demonstrates that the low-resolution draft attention closely approximates the full attention, providing reliable guidance for constructing accurate sparse attention. Experimental results show that our method outperforms existing sparse attention approaches in video generation quality and achieves up to 1.75x end-to-end speedup on GPUs. Code: https://github.com/shawnricecake/draft-attention
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。