不重新训练,让视频扩散模型加速3倍,画质几乎不变。
LoSA: Near-Lossless Sparse Attention for Training-Free Video Diffusion Acceleration

- 用99%注意力质量约束,自动选关键帧块,只算必要计算
- 在万像1.3B上提速1.36倍,画质损失仅0.06分
- 适合追求高画质的视频生成应用,尤其配合特征缓存
视频扩散模型采样成本高昂:每个去噪步骤需对长3D标记序列做自注意力,计算复杂度为二次方,随分辨率和时长增长而主导整体开销。稀疏注意力可降低计算量且无需重训练,但现有方法追求激进稀疏性,速度提升代价是注意力保真度大幅下降。本文反其道而行之:通过构造保证近无损保真度,再在该约束下尽可能减少计算。两个发现使其可行:约40%的块间交互可被移除,仍保留99%注意力质量;高注意力质量区域在去噪过程中保持稳定。提出LoSA,一种无需训练的稀疏注意力方法:设定保留注意力质量99%而非稀疏率,先在早期稠密步骤精确测量各头与查询块的注意力质量,保留满足阈值的最小关键/值块集合,并将这些冻结的块索引复用于后续所有步骤。在Wan2.1-1.3B上,仅用LoSA即实现1.36倍加速,总体评分(VBench)下降0.06点。在组合场景中,结合特征缓存后,在HunyuanVideo上达到3.2倍加速,仅下降0.02点,而最强稀疏基线在类似速度下下降0.32点。在三个视频扩散变压器上,加速比达3.2倍,LoSA始终取得最优的免训练速度-质量权衡。
原文摘要 · Abstract (English)
Video diffusion transformers are costly to sample: every denoising step applies self-attention over a long 3D token sequence, a quadratic cost that dominates as resolution and duration grow. Sparse attention reduces this cost without retraining, but existing methods pursue aggressive sparsity, where further speedup costs disproportionately more attention fidelity. We target the opposite end of this trade-off: fix near-lossless fidelity by construction, and remove as much computation as this constraint permits. Two observations make this regime practical: roughly 40% of block interactions can be removed while retaining 99% of the attention mass, and the high-mass support remains stable across denoising steps. We propose LoSA, a training-free sparse-attention method that fixes a retained-mass threshold of 99% rather than a sparsity ratio: it measures exact block attention masses at one early dense step, keeps, for each head and query block, the smallest key/value block set meeting the threshold, and reuses the frozen block indices for all remaining steps. On Wan2.1-1.3B, LoSA alone gives a $1.36\times$ speedup with a 0.06-point VBench Overall drop. The benefit is largest under composition: combined with feature caching, LoSA reaches a $3.2\times$ speedup on HunyuanVideo at a 0.02-point drop, versus 0.32 points for the strongest sparse baseline at comparable speed. Across three video diffusion transformers and speedups up to $3.2\times$, LoSA consistently achieves the best training-free speed-quality trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。