提出高效长文本推理的稀疏注意力机制,精度不降反升。
Adamas: Hadamard Sparse Attention for Efficient Long-Context Inference
- 用哈达玛变换+分桶+2比特压缩生成紧凑表示,快速选最相关键值对。
- 仅用64个令牌预算即可达到全注意力精度,128个时近乎无损。
- 适合需要超长上下文的文档摘要、代码生成等场景,提速最高4.4倍。
大型语言模型现已支持数万至百万级标记的上下文窗口,适用于长文档摘要、大规模代码生成、多文档问答和持续多轮对话等应用。然而,扩展上下文加剧了自回归解码中自注意力的二次计算开销,导致严重延迟。现有稀疏注意力方法虽缓解成本,但依赖启发式模式,难以召回关键键值对,造成精度下降。本文提出Adamas,一种轻量且高精度的稀疏注意力机制,专为长上下文推理设计。Adamas结合哈达玛变换、分桶与2比特压缩生成紧凑表示,并利用曼哈顿距离估计实现高效的top-k选择。实验表明,Adamas在仅64令牌预算下即可匹配全注意力精度,128令牌时近乎无损,支持的稀疏度比当前最优方法高出8倍,32K序列下自注意力速度提升最高4.4倍,端到端速度提升1.5倍。尤为显著的是,Adamas在激进稀疏下仍能达到与全注意力相当甚至更低的困惑度,证明其在保持精度方面的有效性。
原文摘要 · Abstract (English)
Large language models (LLMs) now support context windows of hundreds of thousands to millions of tokens, enabling applications such as long-document summarization, large-scale code synthesis, multi-document question answering and persistent multi-turn dialogue. However, such extended contexts exacerbate the quadratic cost of self-attention, leading to severe latency in autoregressive decoding. Existing sparse attention methods alleviate these costs but rely on heuristic patterns that struggle to recall critical key-value (KV) pairs for each query, resulting in accuracy degradation. We introduce Adamas, a lightweight yet highly accurate sparse attention mechanism designed for long-context inference. Adamas applies the Hadamard transform, bucketization and 2-bit compression to produce compact representations, and leverages Manhattan-distance estimation for efficient top-k selections. Experiments show that Adamas matches the accuracy of full attention with only a 64-token budget, achieves near-lossless performance at 128, and supports up to 8x higher sparsity than prior state-of-the-art (SOTA) methods while delivering up to 4.4x self-attention and 1.5x end-to-end speedups on 32K-length sequences. Remarkably, Adamas attains comparable or even lower perplexity than full attention, underscoring its effectiveness in maintaining accuracy under aggressive sparsity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。