通过利用扩散过程中的时间一致性，实现高效视频生成注意力计算。
LiteAttention: A Temporal Sparse Attention for Diffusion Transformers
- 基于时间连贯性，提前标记冗余计算块并持续跳过。
- 在不降质前提下实现显著加速，支持生产级视频生成模型。
- 兼具动态自适应与静态高效优势，适合大规模视频生成任务。
扩散变换器在视频生成中表现卓越，但其注意力计算复杂度为二次方，导致延迟过高。现有加速方法面临根本矛盾：动态稀疏模式需每步估算，带来高开销和误差；静态模式固定不变，常不最优。本文发现扩散注意力的稀疏模式在去噪步骤间具有强时间一致性——某时刻非关键的图像块在后续步骤仍保持非关键。基于此，提出LiteAttention，通过早期标记非必要块并向前传播跳过决策，避免重复探查开销，实现跨去噪序列的演化式计算跳过。该方法结合了动态方法的自适应性与静态方法的高效性。我们在FlashAttention基础上实现高度优化的LiteAttention内核，在生产级视频扩散模型上取得显著提速，且质量无损失。代码与实现细节将公开。
原文摘要 · Abstract (English)
Diffusion Transformers, particularly for video generation, achieve remarkable quality but suffer from quadratic attention complexity, leading to prohibitive latency. Existing acceleration methods face a fundamental trade-off: dynamically estimating sparse attention patterns at each denoising step incurs high computational overhead and estimation errors, while static sparsity patterns remain fixed and often suboptimal throughout denoising. We identify a key structural property of diffusion attention, namely, its sparsity patterns exhibit strong temporal coherence across denoising steps. Tiles deemed non-essential at step $t$ typically remain so at step $t+δ$. Leveraging this observation, we introduce LiteAttention, a method that exploits temporal coherence to enable evolutionary computation skips across the denoising sequence. By marking non-essential tiles early and propagating skip decisions forward, LiteAttention eliminates redundant attention computations without repeated profiling overheads, combining the adaptivity of dynamic methods with the efficiency of static ones. We implement a highly optimized LiteAttention kernel on top of FlashAttention and demonstrate substantial speedups on production video diffusion models, with no degradation in quality. The code and implementation details will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。