通过挖掘视频注意力的结构化稀疏性，实现超长视频生成加速
Compact Attention: Exploiting Structured Spatio-Temporal Sparsity for Fast Video Generation
- 动态分块与可变时窗结合，自适应捕捉不同空间模式
- 单卡下注意力计算提速1.6~2.5倍，画质与全注意力相当
- 适合追求高效长视频生成的研究者和开发者
基于Transformer的视频生成面临自注意力机制的高计算开销，尤其在生成超长序列时更为突出。现有方法如分解注意力和固定稀疏模式，未能充分挖掘视频数据中的时空冗余。通过对视频扩散变换器（DiT）的系统分析，我们发现注意力矩阵具有结构化但异质的稀疏模式，特定注意力头会动态关注不同的时空区域（如局部、十字形或全局模式）。当前稀疏方法要么约束过强，要么引入显著开销。为此，我们提出Compact Attention，一个硬件感知的加速框架，包含三项创新：1）自适应分块策略，通过动态分组近似多样的空间交互模式；2）随帧距变化的时间窗口，调节稀疏程度；3）自动化配置搜索算法，在保留关键注意力路径的同时优化稀疏模式。该方法在单卡环境下实现1.6~2.5倍的注意力计算加速，同时保持与全注意力基线相当的视觉质量。本工作为通过结构化稀疏实现高效长视频生成提供了原则性方案。
原文摘要 · Abstract (English)
The computational demands of self-attention mechanisms pose a critical challenge for transformer-based video generation, particularly in synthesizing ultra-long sequences. Current approaches, such as factorized attention and fixed sparse patterns, fail to fully exploit the inherent spatio-temporal redundancies in video data. Through systematic analysis of video diffusion transformers (DiT), we uncover a key insight: Attention matrices exhibit structured, yet heterogeneous sparsity patterns, where specialized heads dynamically attend to distinct spatiotemporal regions (e.g., local pattern, cross-shaped pattern, or global pattern). Existing sparse attention methods either impose rigid constraints or introduce significant overhead, limiting their effectiveness. To address this, we propose Compact Attention, a hardware-aware acceleration framework featuring three innovations: 1) Adaptive tiling strategies that approximate diverse spatial interaction patterns via dynamic tile grouping, 2) Temporally varying windows that adjust sparsity levels based on frame proximity, and 3) An automated configuration search algorithm that optimizes sparse patterns while preserving critical attention pathways. Our method achieves 1.6~2.5x acceleration in attention computation on single-GPU setups while maintaining comparable visual quality with full-attention baselines. This work provides a principled approach to unlocking efficient long-form video generation through structured sparsity exploitation. Project Page: https://yo-ava.github.io/Compact-Attention.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。