通过重塑注意力分布，实现超稀疏视觉生成且保持高质量。
Re-ttention: Ultra Sparse Visual Generation via Attention Statistical Reshape
- 利用扩散模型的时间冗余性重塑注意力得分分布
- 推理时仅需3.1%的令牌，仍保持高质量生成
- 适合追求高效推理的视频/图像生成应用
扩散Transformer（DiT）已成为生成高质量视觉内容（如视频和图像）的主流模型。其主要瓶颈在于注意力机制，复杂度随分辨率和视频长度呈二次增长。稀疏注意力虽可缓解此问题，但现有方法在极高稀疏度下难以保持视觉质量，甚至引入显著计算开销。为此，本文提出Re-ttention，通过利用扩散模型的时间冗余性，克服注意力机制中的概率归一化偏移问题，实现极高的稀疏注意力。具体而言，Re-ttention基于先前softmax分布的历史信息重塑注意力分数，从而在极高稀疏度下保持全二次注意力的视觉质量。在CogVideoX和PixArt DiTs等T2V/T2I模型上的实验表明，Re-ttention推理时仅需3.1%的令牌，优于FastDiTAttn、Sparse VideoGen和MInference等当前方法。
原文摘要 · Abstract (English)
Diffusion Transformers (DiT) have become the de-facto model for generating high-quality visual content like videos and images. A huge bottleneck is the attention mechanism where complexity scales quadratically with resolution and video length. One logical way to lessen this burden is sparse attention, where only a subset of tokens or patches are included in the calculation. However, existing techniques fail to preserve visual quality at extremely high sparsity levels and might even incur non-negligible compute overheads. To address this concern, we propose Re-ttention, which implements very high sparse attention for visual generation models by leveraging the temporal redundancy of Diffusion Models to overcome the probabilistic normalization shift within the attention mechanism. Specifically, Re-ttention reshapes attention scores based on the prior softmax distribution history in order to preserve the visual quality of the full quadratic attention at very high sparsity levels. Experimental results on T2V/T2I models such as CogVideoX and the PixArt DiTs demonstrate that Re-ttention requires as few as 3.1% of the tokens during inference, outperforming contemporary methods like FastDiTAttn, Sparse VideoGen and MInference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。