用线性注意力替换部分模块,让视频生成快1.25倍且不丢质量
LinVideo: A Post-Training Framework towards O(n) Attention in Efficient Video Generation
- 自动识别可替换层,逐步转为线性注意力
- 采样轨迹分布匹配,恢复模型性能
- 无需训练数据,适合高效视频生成场景
视频扩散模型虽能生成高质量视频,但自注意力机制导致计算成本随序列长度呈平方增长。尽管线性注意力可降低开销,但完全替代需昂贵预训练,因线性注意力表达能力有限且视频时空建模复杂。本文提出LinVideo,一种无需数据的后训练框架,将指定数量的自注意力模块替换为线性注意力,同时保持原模型性能。我们发现不同层替换难易程度差异显著,将层选择建模为二分类问题,提出选择性迁移策略,可自动、渐进地转换层,影响极小。此外,针对现有目标在迁移过程中的低效与无效问题,引入任意时间分布匹配(ADM)目标,对采样轨迹中任意时刻的样本分布进行对齐,高效恢复性能。大量实验表明,该方法实现1.25-2.00倍加速,4步蒸馏模型更达15.92倍延迟降低,视觉质量损失极小。
原文摘要 · Abstract (English)
Video diffusion models (DMs) have enabled high-quality video synthesis. However, their computation costs scale quadratically with sequence length because self-attention has quadratic complexity. While linear attention lowers the cost, fully replacing quadratic attention requires expensive pretraining due to the limited expressiveness of linear attention and the complexity of spatiotemporal modeling in video generation. In this paper, we present LinVideo, an efficient data-free post-training framework that replaces a target number of self-attention modules with linear attention while preserving the original model's performance. First, we observe a significant disparity in the replaceability of different layers. Instead of manual or heuristic choices, we frame layer selection as a binary classification problem and propose selective transfer, which automatically and progressively converts layers to linear attention with minimal performance impact. Additionally, to overcome the ineffectiveness and inefficiency of existing objectives for this transfer process, we introduce an anytime distribution matching (ADM) objective that aligns the distributions of samples across any timestep along the sampling trajectory. This objective is efficient and recovers model performance. Extensive experiments show that our method achieves a 1.25-2.00x speedup while preserving generation quality, and our 4-step distilled model further delivers a 15.92x latency reduction with minimal visual quality drop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。