arXiv:2605.19726cs.CV2026-05被引 1

提出块级稀疏注意力机制,显著提升长文本生成模型的效率与稳定性。

Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention

论文配图:Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention
图 1 · 摘自论文原文
  • 通过下采样空间识别关键信息区域,避免依赖固定位置先验。
  • 在50%稀疏度下保持接近全注意力性能,计算速度提升6.95倍。
  • 适用于语言、多模态及视频生成模型,通用性强且适合长序列任务。

扩散语言模型(DLMs)实现全局连贯、双向且可控的文本生成,相较于传统自回归大模型具有优势,但扩展至超长序列仍成本高昂。现有块稀疏注意力方法依赖高分辨率注意力空间中的固定采样模式(如尾部区域或反对角线条带),此类基于先验的采样可能遗漏重要标记,并在分布变化时引入不稳定性。本文提出块近似稀疏注意力框架(BA-Att),采用块级预下采样操作,在紧凑的下采样空间中识别信息丰富区域,避免对脆弱位置先验的依赖。为分析其理论行为,定义了理想后下采样注意力图,并形式化了预下采样与后下采样方案间的近似误差。基于此,引入轻量级范数排序模块和协方差补偿校正,通过仅用QK方差对角线近似完整协方差,降低计算复杂度。大量实验表明,该算子在注意力计算上相比FlashAttention最高加速6.95倍,且在语言模型、多模态语言模型及视频生成模型中,于50%稀疏度下维持接近全注意力性能,展现出强高效性与泛化能力。

原文摘要 · Abstract (English)

Diffusion Language Models (DLMs) enable globally coherent, bidirectional, and controllable text generation, offering advantages over traditional autoregressive LLMs, while scaling to ultra-long sequences remains costly. Many existing block-sparse attention methods select blocks by fixed sampling patterns over the high-resolution attention space, such as tail regions or anti-diagonal stripes. Such prior-driven sampling can miss salient tokens and introduce instability under distribution shifts. In this paper, we propose the Block Approximate Sparse Attention framework (BA-Att) with block-wise pre-downsampled operation, which identifies informative regions within a compact downsampled space, avoiding reliance on brittle positional priors. To analyze its theoretical behavior, we define an oracle post-downsample attention map and formalize the approximation error between pre- and post-downsample schemes. Based on this insight, we introduce a lightweight norm-sorting module and a covariance-compensated correction that approximates full covariance using diagonal QK variances, reducing computational complexity. Extensive experiments show that our operator achieves up to 6.95x acceleration over FlashAttention in attention computation, and maintains near full-attention performance at 50% sparsity across language models, multimodal language models, and video generation models, demonstrating strong efficiency and generalization.

扩散模型稀疏注意力长序列生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。