提出径向注意力机制,让长视频生成速度提升近4倍且成本大幅降低。
Radial Attention: $O(n\log n)$ Sparse Attention with Energy Decay for Long Video Generation
- 用静态掩码设计稀疏注意力,随时间距离缩小关注范围,计算量降为O(n log n)。
- 在多个模型上实现最高1.9倍推理加速,训练成本降低最多4.4倍,生成长度扩展至4倍。
- 兼容现有模型,仅需少量微调即可拓展长视频生成能力,适合高效部署场景。
扩散模型虽已实现高质量视频生成,但时间维度带来巨大计算开销,使长视频训练与推理成本高昂。本文发现视频扩散模型中存在时空能量衰减现象:后softmax注意力分数随空间与时间距离增加而衰减,类似自然信号传播规律。受此启发,提出径向注意力(Radial Attention),一种复杂度为O(n log n)的可扩展稀疏注意力机制,将能量衰减转化为计算密度的指数衰减,显著优于标准O(n²)密集注意力,也比线性注意力更富表达力。该方法采用简单静态注意力掩码,每个令牌仅关注空间邻近区域,且关注窗口随时间距离递减。此外,支持预训练视频扩散模型通过高效的LoRA微调扩展生成长度。大量实验表明,该方法在Wan2.1-14B、HunyuanVideo和Mochi 1等模型上保持视频质量,相比原密集注意力实现最高1.9倍推理加速;无需大量调参即可将生成长度扩展至4倍,训练成本降低最多4.4倍,推理速度提升最高3.7倍。代码已开源。
原文摘要 · Abstract (English)
Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training and inference on long videos prohibitively expensive. In this paper, we identify a phenomenon we term Spatiotemporal Energy Decay in video diffusion models: post-softmax attention scores diminish as spatial and temporal distance between tokens increase, akin to the physical decay of signal or waves over space and time in nature. Motivated by this, we propose Radial Attention, a scalable sparse attention mechanism with $\mathcal{O}(n \log n)$ complexity that translates energy decay into exponentially decaying compute density, which is significantly more efficient than standard $\mathcal{O}(n^2)$ dense attention and more expressive than linear attention. Specifically, Radial Attention employs a simple, static attention mask where each token attends to spatially nearby tokens, with the attention window size shrinking with temporal distance. Moreover, it allows pre-trained video diffusion models to extend their generation length with efficient LoRA-based fine-tuning. Extensive experiments show that Radial Attention maintains video quality across Wan2.1-14B, HunyuanVideo, and Mochi 1, achieving up to a 1.9$\times$ speedup over the original dense attention. With minimal tuning, it enables video generation up to 4$\times$ longer while reducing training costs by up to 4.4$\times$ compared to direct fine-tuning and accelerating inference by up to 3.7$\times$ compared to dense attention inference. Code is released at \href{https://github.com/mit-han-lab/radial-attention}{https://github.com/mit-han-lab/radial-attention}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。