通过轻量线性注意力实现90%稀疏度，显著提速视频生成模型。
SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer
- 并行引入线性注意力分支，用多级动静态缩放策略平衡性能与效率。
- 最高达90%稀疏度，推理速度提升1.52至2.03倍，生成质量接近全注意力基线。
- 仅需2000张视频样本、不足1600步微调，30小时GPU即可完成训练。
扩散Transformer在视频生成中表现优异，但其长序列输入导致注意力计算呈二次复杂度，带来巨大延迟。现有稀疏注意力方法中，无需训练的方法稀疏度有限，加速效果有限；而基于训练的方法虽可实现高稀疏度，却需大量数据和计算资源。本文提出SALAD，引入一个轻量级线性注意力分支，与稀疏注意力并行，并采用多级静态-动态缩放策略平衡两者。该方法在不同模型和序列长度下实现高达90%的稀疏度，推理速度提升1.52至2.03倍，同时保持与全注意力基线相当的生成质量。此外，微调过程高效：仅需2,000个视频样本、少于1,600次训练步，且不超过30个GPU小时（批量大小为8）。
原文摘要 · Abstract (English)
Diffusion Transformers have demonstrated remarkable performance in video generation. However, their long input sequences incur substantial latency due to the quadratic complexity of full attention. Various sparse attention mechanisms have been proposed. Training-free approaches are limited to moderate sparsity and thus yield only modest acceleration, whereas training-based methods can reach much higher sparsity but demand substantial data and computation. In this work, we propose SALAD, introducing a lightweight linear attention branch in parallel with the sparse attention. Leveraging a Multi-level Static-Dynamic Scaling Strategy to balance the two branches, our method attains up to 90% sparsity and 1.52-2.03x inference speedup across different models and sequence lengths, while maintaining generation quality comparable to the full attention baseline. Moreover, our finetuning process is highly efficient, requiring only 2,000 video samples, fewer than 1,600 training steps, and no more than 30 GPU hours with a batch size of 8.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。