让视频生成模型更快：用轻量改造实现线性注意力，不重训练也能提速。
Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer
- 混合软最大化与线性注意力，保持表达力的同时降低计算开销。
- 在瓦尼2.1模型上应用后，长视频生成延迟下降40%以上，显存减少35%。
- 仅需几天GPU时间微调，适合想加速现有视频模型的开发者使用。
基于Transformer的视频扩散模型（VDM）虽生成质量顶尖，但自注意力机制的二次方复杂度限制了长序列与高分辨率生成的效率。尽管线性注意力可降低复杂度，以往方法需重训练且难以匹配软最大化注意力的表达能力。本文提出Attention Surgery，一种高效框架，可在预训练VDM中直接引入线性或混合注意力，无需从头训练。受语言模型启发，该方法结合新型混合注意力机制——融合软最大化与线性注意力的令牌——并辅以轻量级知识蒸馏与微调流程，仅需数天GPU时间。同时引入成本感知的分块率策略，在各层间平衡表达力与效率。应用于瓦尼2.1 1.3B模型，在VBench、VBench2.0及人工偏好评估中表现优异。移动端实测显示，长视频生成的延迟、内存占用与浮点运算量均有显著优化，扩展性明显提升。
原文摘要 · Abstract (English)
Transformer-based video diffusion models (VDMs) deliver state-of-the-art video generation quality but are constrained by the quadratic cost of self-attention, making long sequences and high resolutions computationally expensive. While linear attention offers sub-quadratic complexity, previous approaches have failed to match the expressiveness of softmax attention unless retrained at significant computational cost. We introduce Attention Surgery, an efficient framework that enables linear or hybrid attention in pretrained VDMs, eliminating the need for training from scratch. Inspired by recent advances in language models, our method combines a novel hybrid attention mechanism-mixing softmax and linear tokens-with a lightweight distillation and fine-tuning pipeline requiring only a few GPU-days. Additionally, we incorporate a cost-aware block-rate strategy to balance expressiveness and efficiency across layers. Applied to Wan2.1 1.3B, a state-of-the-art efficient transformer VDM and evaluated on VBench, VBench2.0 and a human preference study, Attention Surgery achieves competitive results. Furthermore, measurements of on-mobile latency, memory usage, and FLOPs demonstrate notable improvements in scaling behavior for longer videos. Project page is available at: https://qualcomm-ai-research.github.io/attention-surgery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。