动态选择重要注意力项，提升长文本大模型推理速度与精度。
Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
- 根据累计注意力得分自适应选词，无需固定数量限制。
- 相比现有方法，解码速度提升7.29倍，端到端提速1.58倍。
- 适合对精度要求高的长文本生成任务，如法律、医疗文档处理。
长上下文模型在诸多应用中至关重要，但解码时加载大型键值缓存效率低下。已有方法对稀疏注意力采用固定令牌预算，假设固定数量的令牌可近似全注意力，却忽略了不同头、层和上下文中注意力重要性的差异。为此，我们提出Tactic，一种自适应稀疏注意力机制，不依赖固定预算，而是基于累积注意力得分动态选择令牌。通过设定目标注意力分数占比，Tactic能自然适应注意力稀疏性的变化。为高效逼近该选择过程，Tactic采用基于聚类的排序与分布拟合，以极低计算开销精准估计令牌重要性。实验表明，Tactic优于现有稀疏注意力算法，在保持更高精度的同时，实现最高达7.29倍的解码注意力加速，整体推理速度提升1.58倍，是精度敏感型长上下文LLM推理的实用有效方案。
原文摘要 · Abstract (English)
Long-context models are essential for many applications but face inefficiencies in loading large KV caches during decoding. Prior methods enforce fixed token budgets for sparse attention, assuming a set number of tokens can approximate full attention. However, these methods overlook variations in the importance of attention across heads, layers, and contexts. To address these limitations, we propose Tactic, a sparsity-adaptive and calibration-free sparse attention mechanism that dynamically selects tokens based on their cumulative attention scores rather than a fixed token budget. By setting a target fraction of total attention scores, Tactic ensures that token selection naturally adapts to variations in attention sparsity. To efficiently approximate this selection, Tactic leverages clustering-based sorting and distribution fitting, allowing it to accurately estimate token importance with minimal computational overhead. We show that Tactic outperforms existing sparse attention algorithms, achieving superior accuracy and up to 7.29x decode attention speedup. This improvement translates to an overall 1.58x end-to-end inference speedup, making Tactic a practical and effective solution for long-context LLM inference in accuracy-sensitive applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。