arXiv:2605.23445cs.CV2026-05被引 6

提出动态细粒度稀疏注意力,高效生成视频且质量不降。

DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation

论文配图:DFSAttn: Dynamic Fine-grained Sparse Attention for Efficient Video Generation
图 1 · 摘自论文原文
  • 用希尔伯特曲线重排令牌,实现精细稀疏化并适配GPU加速。
  • 在高稀疏率下保持高质量,端到端速度提升2.1倍。
  • 无需训练,适合追求高效视频生成的研究者与开发者。

扩散变压器在高质量视频生成中表现卓越,但其依赖时空3D全注意力导致计算成本高昂,因注意力的二次复杂度。块稀疏注意力通过聚焦重要区域缓解此问题,但扩散变压器中的注意力图具有固有的动态和细粒度稀疏性,使现有块稀疏方法在高稀疏率下质量显著下降。本文重新审视块稀疏注意力,推导出注意力召回率的理论下界,揭示其有效性关键因素。基于此,提出DFSAttn,一种无需训练的稀疏注意力框架,可高效实现动态细粒度稀疏化。DFSAttn包含三项核心设计:基于希尔伯特曲线的令牌重排以实现细粒度稀疏化并保留高效GPU执行;分层块评分用于准确估计块重要性;以及自适应比率的稀疏掩码缓存以平衡精度与效率。实验表明,DFSAttn在高稀疏率下持续优于先前方法,端到端速度最高提升2.1倍,同时保持高质量生成。代码已开源,地址为https://github.com/jessica-hujie/DFSAttn。

原文摘要 · Abstract (English)

Diffusion transformers have achieved remarkable success in high-quality video generation, yet their reliance on spatiotemporal 3D full attention incurs prohibitive computational cost due to the quadratic complexity of attention. Block sparse attention is a common approach to mitigate this by focusing computation on important regions. However, attention maps in DiTs exhibit inherently dynamic and fine-grained sparsity, which causes existing block sparse attention methods to degrade significantly in quality, especially at high sparsity ratios. In this paper, we revisit block sparse attention and derive a theoretical lower bound on attention recall to characterize the key factors governing its effectiveness. Guided by these insights, we propose DFSAttn, a training-free sparse attention framework that enables dynamic, fine-grained sparsification efficiently. DFSAttn incorporates three core designs: Hilbert curve-based token reordering to achieve fine-grained sparsity while preserving efficient GPU execution, hierarchical block scoring for accurate block importance estimation, and sparse mask caching with adaptive ratios to balance accuracy and efficiency. Experimental results demonstrate that DFSAttn consistently outperforms prior methods under high sparsity, achieving up to 2.1$\times$ end-to-end speedup while maintaining high generation quality. Our code is open-sourced and available at https://github.com/jessica-hujie/DFSAttn.

视频生成稀疏注意力扩散模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。