arXiv:2512.12087cs.CL2025-12被引 9

通过动态阈值跳过无关注意力块,显著加速大模型长文本推理。

BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding

  • 用固定阈值动态跳过低贡献的注意力块,无需训练或预计算。
  • 在现代GPU上实现预填充1.52倍、解码1.48倍加速,稀疏度达70%以上。
  • 兼容主流注意力形式,部署简单,适合追求高效推理的工程师。

大型语言模型对长上下文推理的需求加剧了自注意力机制带来的计算与内存瓶颈。为应对这一挑战,我们提出BLASST,一种即插即用的动态稀疏注意力机制,通过单一标量阈值跳过注意力块以加速推理。该方法无需训练,避免昂贵的预计算,可加速所有主流注意力变体(MHA、GQA、MQA、MLA)的预填充与解码阶段,并适配现代硬件,易于集成至现有框架。其核心是复用在线softmax统计信息识别可忽略的注意力分数,从而跳过softmax计算、值块加载及后续矩阵乘法。我们实现了优化内核,延迟开销极小。自动化阈值校准表明,最优阈值与上下文长度呈简单反比关系,每模型仅需预填充和解码各一个阈值。在保持基准准确率前提下,预填充阶段实现1.52倍加速(71.9%稀疏度),解码阶段实现1.48倍加速(73.2%稀疏度),均基于现代GPU验证。

原文摘要 · Abstract (English)

The growing demand for long-context inference capabilities in Large Language Models (LLMs) has intensified the computational and memory bottlenecks inherent to the self-attention mechanism. To address this challenge, we introduce BLASST, a drop-in, dynamic sparse attention mechanism that accelerates inference by using only a fixed scalar threshold to skip attention blocks. Our method targets practical inference deployment by removing the barriers to adoption present in existing works. As such, BLASST eliminates training requirements, avoids expensive pre-computation passes, accelerates both prefill and decode across all major attention variants (MHA, GQA, MQA, and MLA), provides optimized support for modern hardware, and easily integrates into existing frameworks. This is achieved by reusing online softmax statistics to identify negligible attention scores, skipping softmax, value block loads, and the subsequent matrix multiplication. We demonstrate the BLASST algorithm by delivering optimized kernels with negligible latency overhead. Our automated threshold calibration procedure reveals a simple inverse relationship between optimal threshold and context length, meaning we require only a single threshold each for prefill and decode per model. Preserving benchmark accuracy, we demonstrate a 1.52x speedup for prefill at 71.9% sparsity and a 1.48x speedup for decode at 73.2% sparsity on modern GPUs.

注意力稀疏长文本推理模型优化GPU加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。