arXiv:2502.21079cs.CV2025-02ICCV被引 67

提出无需训练的自适应稀疏注意力,显著加速长视频生成。

Training-free and Adaptive Sparse Attention for Efficient Long Video Generation

  • 采用分块动态稀疏模式,捕捉扩散模型中跨模态的层级稀疏结构。
  • 通过输入无关的注意力模式复用,在生成时实时精准定位稀疏索引。
  • 即插即用无额外训练,适用于多种视频生成模型,提速明显且画质不变。

使用扩散变换器(DiTs)生成高保真长视频常受注意力机制计算开销影响,导致显著延迟。例如,生成一个8秒720p视频(110K tokens)需约600 PFLOPs,其中约500 PFLOPs用于注意力计算。为此,我们提出AdaSpa——首个动态模式与在线精准搜索结合的稀疏注意力方法。首先,为实现动态模式,引入分块稀疏模式,高效捕获DiTs中跨模态与模态内固有的层级化稀疏结构;该结构具有分块特性,显著降低注意力复杂度,同时保持高质量输出。其次,为支持在线精准搜索,提出融合LSE缓存的搜索机制与头自适应分层块稀疏注意力,基于观察:尽管注意力稀疏模式和对数似然估计(LSE)随输入、层数与注意力头变化,但其在去噪步骤间保持不变。利用这一不变性,实现低开销的实时稀疏索引精准识别。AdaSpa作为自适应、即插即用方案,可无缝集成现有DiTs,无需额外微调或数据依赖性调优。大量实验表明,该方法在多种模型上均实现显著加速,同时保持视频质量,验证了其在高效视频生成中的鲁棒性与可扩展性。

原文摘要 · Abstract (English)

Generating high-fidelity long videos with Diffusion Transformers (DiTs) is often hindered by significant latency, primarily due to the computational demands of attention mechanisms. For instance, generating an 8-second 720p video (110K tokens) with HunyuanVideo takes about 600 PFLOPs, with around 500 PFLOPs consumed by attention computations. To address this issue, we propose AdaSpa, the first Dynamic Pattern and Online Precise Search sparse attention method. Firstly, to realize the Dynamic Pattern, we introduce a blockified pattern to efficiently capture the hierarchical sparsity inherent in DiTs. This is based on our observation that sparse characteristics of DiTs exhibit hierarchical and blockified structures between and within different modalities. This blockified approach significantly reduces the complexity of attention computation while maintaining high fidelity in the generated videos. Secondly, to enable Online Precise Search, we propose the Fused LSE-Cached Search with Head-adaptive Hierarchical Block Sparse Attention. This method is motivated by our finding that DiTs' sparse pattern and LSE vary w.r.t. inputs, layers, and heads, but remain invariant across denoising steps. By leveraging this invariance across denoising steps, it adapts to the dynamic nature of DiTs and allows for precise, real-time identification of sparse indices with minimal overhead. AdaSpa is implemented as an adaptive, plug-and-play solution and can be integrated seamlessly with existing DiTs, requiring neither additional fine-tuning nor a dataset-dependent profiling. Extensive experiments validate that AdaSpa delivers substantial acceleration across various models while preserving video quality, establishing itself as a robust and scalable approach to efficient video generation.

视频生成稀疏注意力扩散模型效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。