arXiv:2605.20813cs.CL2026-05

通过周期刷新列稀疏注意力,加速扩散语言模型推理。

PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models

论文配图:PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models
图 1 · 摘自论文原文
  • 采用细粒度列稀疏结构替代粗粒度块稀疏,保留关键注意力交互。
  • 在早期去噪步骤中识别并复用稀疏模式,仅在少数中间步刷新,提升效率。
  • 支持多种上下文长度,实测速度比FlashAttention快1.95倍。

扩散大语言模型(dLLMs)的推理计算成本高昂,因去噪过程中每一步都需重复执行全自注意力且无法使用键值缓存。现有稀疏注意力方法通过块稀疏计算缓解此问题,但仅在后期迭代应用,对整体效率提升有限。本文提出PulseCol,一种周期性刷新的列稀疏注意力机制,将粗粒度块级稀疏替换为更精细的列级稀疏结构,更精准保留重要注意力关系,同时暴露更高稀疏性。基于该设计,PulseCol在早期去噪步骤中识别稀疏模式,并在后续迭代中复用,仅在少量中间步骤刷新以跟踪注意力模式演化。实验表明,PulseCol相比已有稀疏方法实现更高稀疏率与更大实际加速,且保持模型质量。借助优化的列稀疏注意力GPU内核,PulseCol在多个上下文长度下相较FlashAttention实现最高1.95×端到端加速。

原文摘要 · Abstract (English)

Inference in diffusion large language models (dLLMs) is computationally expensive, as full self-attention must be repeatedly executed at each step of the denoising process without KV cache. Recent sparse attention methods for dLLMs mitigate this cost via block-sparse computation, which is applied only in later iterations when model performance is less sensitive to coarse-grained sparse approximation, but yields limited improvements in computational efficiency and acceleration. This motivates a finer-grained sparsification strategy that can be applied from earlier iterations and leverages reusable sparsity patterns, enabling further efficiency gains. In this work, we introduce PulseCol, a periodically refreshed column-sparse attention method for accelerating diffusion language models. PulseCol replaces coarse block-level sparsity with a finer-grained column-sparse structure, allowing important attention interactions to be retained more precisely while exposing greater sparsity. Built on this column-level formulation, PulseCol further identifies sparse patterns at the early denoising step and reuses them across subsequent iterations, refreshing them only at a small number of intermediate steps to track the evolution of sparse attention patterns during denoising. Experiments show that PulseCol achieves higher sparsity and greater practical speedup than prior sparse attention methods for dLLMs, while maintaining model quality. Enabled by optimized GPU kernels for column-sparse attention, PulseCol delivers up to 1.95$\times$ end-to-end speedup over FlashAttention across several context lengths.

扩散模型注意力机制稀疏计算推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。