arXiv:2512.24086cs.CV2025-12被引 3

提出轻量级稀疏注意力机制,加速图像视频生成模型且兼容多种硬件。

RainFusion2.0: Temporal-Spatial Awareness and Hardware-Efficient Block-wise Sparse Attention

  • 用块均值代表令牌预测稀疏模式,降低计算开销。
  • 实现时空感知的令牌重排,支持80%稀疏度下1.5~1.8倍加速。
  • 专为视频生成设计首帧锚点机制,跨硬件平台通用性强。

在图像与视频生成任务中,扩散变换器(DiT)模型因注意力机制带来极高计算成本,限制了实际应用。随着硬件发展,除GPU外,专用集成电路(ASIC)等设备越来越多用于模型推理。稀疏注意力通过跳过不重要令牌的计算来缓解计算压力,但现有方法存在稀疏模式预测开销大、缺乏硬件通用性等问题,多数仅针对GPU设计。为此,本文提出RainFusion2.0,构建一种在线自适应、硬件高效、低开销的稀疏注意力机制,可加速视频与图像生成模型,并在多种硬件平台上保持鲁棒性能。关键技术包括:(1) 利用块内均值作为代表性令牌进行稀疏掩码预测;(2) 实现时空感知的令牌重排;(3) 针对视频生成场景引入首帧锚点机制。实验表明,RainFusion2.0可在保持视频质量的前提下实现80%稀疏度,并带来1.5~1.8倍端到端加速。该方法在多种生成模型中有效,且具备跨硬件平台的良好泛化能力。

原文摘要 · Abstract (English)

In video and image generation tasks, Diffusion Transformer (DiT) models incur extremely high computational costs due to attention mechanisms, which limits their practical applications. Furthermore, with hardware advancements, a wide range of devices besides graphics processing unit (GPU), such as application-specific integrated circuit (ASIC), have been increasingly adopted for model inference. Sparse attention, which leverages the inherent sparsity of attention by skipping computations for insignificant tokens, is an effective approach to mitigate computational costs. However, existing sparse attention methods have two critical limitations: the overhead of sparse pattern prediction and the lack of hardware generality, as most of these methods are designed for GPU. To address these challenges, this study proposes RainFusion2.0, which aims to develop an online adaptive, hardware-efficient, and low-overhead sparse attention mechanism to accelerate both video and image generative models, with robust performance across diverse hardware platforms. Key technical insights include: (1) leveraging block-wise mean values as representative tokens for sparse mask prediction; (2) implementing spatiotemporal-aware token permutation; and (3) introducing a first-frame sink mechanism specifically designed for video generation scenarios. Experimental results demonstrate that RainFusion2.0 can achieve 80% sparsity while achieving an end-to-end speedup of 1.5~1.8x without compromising video quality. Moreover, RainFusion2.0 demonstrates effectiveness across various generative models and validates its generalization across diverse hardware platforms.

稀疏注意力视频生成硬件高效扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。