通过挖掘视频冗余特性,实现无需训练的加速生成。
RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy
- 利用空间、时间、纹理三类视觉稀疏模式,动态识别注意力区域。
- 推理时计算速度提升超2倍,视频质量损失仅0.2个VBench分。
- 即插即用,兼容主流3D注意力生成模型,无额外训练成本。
使用扩散模型进行视频生成计算开销巨大,其中扩散变换器(DiT)模型的3D注意力机制占总计算资源的80%以上。本文提出 {f RainFusion},一种无需训练的稀疏注意力方法,利用视觉数据固有的稀疏性,在不牺牲视频质量的前提下加速注意力计算。具体而言,我们识别出视频生成注意力计算中的三种独特稀疏模式——空间模式、时间模式和纹理模式。每个注意力头的稀疏模式通过所提出的 {f ARM}(自适应识别模块)在线确定,推理开销极低(约0.2%)。RainFusion为即插即用方案,可无缝集成至领先3D注意力视频生成模型中,无需额外训练或校准。我们在HunyuanVideo、OpenSoraPlan-1.2和CogVideoX-5B等主流开源模型上评估,验证其广泛适用性与有效性。实验表明,RainFusion在保持视频质量的同时,注意力计算速度提升超过2倍,对VBench评分影响仅为-0.2%。
原文摘要 · Abstract (English)
Video generation using diffusion models is highly computationally intensive, with 3D attention in Diffusion Transformer (DiT) models accounting for over 80\% of the total computational resources. In this work, we introduce {\bf RainFusion}, a novel training-free sparse attention method that exploits inherent sparsity nature in visual data to accelerate attention computation while preserving video quality. Specifically, we identify three unique sparse patterns in video generation attention calculations--Spatial Pattern, Temporal Pattern and Textural Pattern. The sparse pattern for each attention head is determined online with negligible overhead (\textasciitilde\,0.2\%) with our proposed {\bf ARM} (Adaptive Recognition Module) during inference. Our proposed {\bf RainFusion} is a plug-and-play method, that can be seamlessly integrated into state-of-the-art 3D-attention video generation models without additional training or calibration. We evaluate our method on leading open-sourced models including HunyuanVideo, OpenSoraPlan-1.2 and CogVideoX-5B, demonstrating its broad applicability and effectiveness. Experimental results show that RainFusion achieves over {\bf 2\(\times\)} speedup in attention computation while maintaining video quality, with only a minimal impact on VBench scores (-0.2\%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。