arXiv:2601.11641cs.CVcs.LG2026-01被引 5

提出无需采样的动态稀疏注意力,提升视频生成效率与质量

Mixture of Distributions Matters: Dynamic Sparse Attention for Efficient Video Diffusion Transformers

  • 用分布混合建模注意力模式,实现无采样动态稀疏
  • 在多个基准上加速同时保持生成质量,性能稳定提升
  • 适合追求高效高质视频生成的开发者和研究者

尽管扩散变换器(DiTs)在视频生成中取得显著进展,但长序列生成仍受自注意力机制固有的二次复杂度制约,成为实际部署的主要障碍。现有稀疏注意力方法或依赖过于简化的静态模式,或需昂贵采样操作实现动态稀疏,导致模式预测不准确、生成质量下降。为此,我们提出一种无采样动态注意力框架——混合分布扩散变换器(MOD-DiT),通过两阶段过程精准建模演变中的注意力模式:首先利用早期去噪步骤的先验信息,采用分布式混合方法构建高效的线性近似模型,用于预测特定去噪区间的掩码模式;其次,采用在线块掩码策略动态应用预测掩码,同时保留历史稀疏信息,避免重复采样。大量实验表明,MOD-DiT在多个基准和模型架构上均实现持续加速与质量提升,验证了其在高效高质量视频生成中的有效性,克服了传统稀疏注意力的计算局限。

原文摘要 · Abstract (English)

While Diffusion Transformers (DiTs) have achieved notable progress in video generation, this long-sequence generation task remains constrained by the quadratic complexity inherent to self-attention mechanisms, creating significant barriers to practical deployment. Although sparse attention methods attempt to address this challenge, existing approaches either rely on oversimplified static patterns or require computationally expensive sampling operations to achieve dynamic sparsity, resulting in inaccurate pattern predictions and degraded generation quality. To overcome these limitations, we propose a \underline{\textbf{M}}ixture-\underline{\textbf{O}}f-\underline{\textbf{D}}istribution \textbf{DiT} (\textbf{MOD-DiT}), a novel sampling-free dynamic attention framework that accurately models evolving attention patterns through a two-stage process. First, MOD-DiT leverages prior information from early denoising steps and adopts a {distributed mixing approach} to model an efficient linear approximation model, which is then used to predict mask patterns for a specific denoising interval. Second, an online block masking strategy dynamically applies these predicted masks while maintaining historical sparsity information, eliminating the need for repetitive sampling operations. Extensive evaluations demonstrate consistent acceleration and quality improvements across multiple benchmarks and model architectures, validating MOD-DiT's effectiveness for efficient, high-quality video generation while overcoming the computational limitations of traditional sparse attention approaches.

视频生成扩散模型稀疏注意力Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。