提出双向稀疏注意力机制，让视频生成模型训练快20倍且不丢质量。
Bidirectional Sparse Attention for Faster Video Diffusion Training
- 动态筛选关键查询与键值对，减少冗余计算。
- 训练速度提升17.79倍，计算量降低20倍。
- 适合追求高效长视频生成的开发者与研究者。
视频扩散变换器（DiT）在生成质量上表现优异，但生成高分辨率、长时长视频时面临严重计算瓶颈。全注意力机制的二次复杂度导致训练与推理成本过高。问题源于两个方面：查询与键值对本质稀疏导致计算浪费，以及固定稀疏模式无法利用DiT的动态注意力特性。为此，我们提出双向稀疏注意力（BSA）框架，首次在3D全注意力中动态稀疏化查询与键值对，显著提升训练与推理效率。BSA通过两个核心组件实现：基于语义相似性与动态时空训练策略选择最具信息量的查询令牌；通过统计动态阈值保留最显著的键值块进行计算。大量实验表明，BSA在长序列上大幅加速DiT训练，最高降低20倍浮点运算量，实现17.79倍的注意力训练速度提升，同时保持甚至超越全注意力的生成质量。
原文摘要 · Abstract (English)
Video diffusion Transformer (DiT) models excel in generative quality but hit major computational bottlenecks when producing high-resolution, long-duration videos. The quadratic complexity of full attention leads to prohibitively high training and inference costs. Full attention inefficiency stems from two key challenges: excessive computation due to the inherent sparsity of Queries and Key-Value pairs, and redundant computation as fixed sparse patterns fail to leverage DiT's dynamic attention. To overcome this limitation, we propose a Bidirectional Sparse Attention (BSA) framework for faster video DiT training, the first to dynamically sparsify both Queries and Key-Value pairs within 3D full attention, thereby substantially improving training and inference efficiency. BSA addresses these issues through two key components. Query sparsity is optimized by selecting the most informative query tokens via semantic similarity and with a dynamic spatial-time training strategy, while KV sparsity is achieved by computing a statistical dynamic threshold to retain only the most salient KV blocks for computation. Extensive experiments demonstrate that BSA significantly accelerates DiT training across long sequences, reducing FLOPs by up to 20x and achieving 17.79x faster attention training, while preserving or even surpassing the generative quality of full attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。