通过识别可跳过的注意力连接，加速文本生成视频的推理速度。
Accelerating Text-to-Video Generation with Calibrated Sparse Attention
- 发现大量注意力连接在不同输入下得分极低且模式重复，可安全跳过。
- 离线校准后生成稀疏注意力模式，推理时实现最多1.58倍提速。
- 无需训练，适用于多种视频生成模型，保持生成质量与对齐效果。
近期扩散模型虽能生成高质量视频，但运行速度慢。这类模型中的大型Transformer主干网络受限于时空注意力计算。本文发现：大量标记间连接在不同输入下得分始终极低，且其模式在查询间重复出现，因此可跳过计算而几乎不影响结果。该现象在局部标记块间同样成立。受此启发，我们提出CalibAtt——一种无需训练的校准稀疏注意力方法。该方法通过离线校准，识别出跨输入稳定的块级稀疏性与重复模式，并为每一层、每头及扩散时间步编译优化后的注意力操作。推理时，仅密集计算选中的依赖关系，其余则以硬件高效方式跳过。在Wan 2.1 14B、Mochi 1及若干少步蒸馏模型上，多分辨率实验表明，CalibAtt实现最高1.58倍端到端加速，优于现有无训练方法，同时保持视频生成质量与文本-视频对齐。
原文摘要 · Abstract (English)
Recent diffusion models enable high-quality video generation, but suffer from slow runtimes. The large transformer-based backbones used in these models are bottlenecked by spatiotemporal attention. In this paper, we identify that a significant fraction of token-to-token connections consistently yield negligible scores across various inputs, and their patterns often repeat across queries. Thus, the attention computation in these cases can be skipped with little to no effect on the result. This observation continues to hold for connections among local token blocks. Motivated by this, we introduce CalibAtt, a training-free method that accelerates video generation via calibrated sparse attention. CalibAtt performs an offline calibration pass that identifies block-level sparsity and repetition patterns that are stable across inputs, and compiles these patterns into optimized attention operations for each layer, head, and diffusion timestep. At inference time, we compute the selected input-dependent connections densely, and skip the unselected ones in a hardware-efficient manner. Extensive experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models at various resolutions show that CalibAtt achieves up to 1.58x end-to-end speedup, outperforming existing training-free methods while maintaining video generation quality and text-video alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。