通过共享精确注意力模式，加速长文本大模型预填充阶段。
Accelerating Prefilling for Long-Context LLMs via Sparse Pattern Sharing
- 利用头间注意力模式高度相似性，共享计算结果
- 仅需对少量头进行全注意力计算，实现高速预填充
- 兼顾速度与精度，适合长文本推理场景
稀疏注意力方法通过利用注意力的固有稀疏性来加速长上下文推理中的预填充阶段，缓解全注意力计算带来的二次复杂度问题。现有方法依赖预设模式或不准确估计来近似注意力行为，难以充分捕捉真实注意力动态，导致效率下降和精度损失。本文提出一种高精度稀疏注意力机制，通过在不同注意力头之间共享相似且精确的注意力模式，更真实地模拟注意力的动态变化。该方法基于两个关键观察：(1) 注意力模式在不同头间具有强相似性；(2) 此相似性在多种输入下仍保持高度稳定。通过有策略地共享已计算出的精确模式，仅需对少数头执行全注意力计算，即可有效捕获实际注意力模式。全面评估表明，本方法在速度上优于或媲美当前最优方法，同时实现最佳整体精度。
原文摘要 · Abstract (English)
Sparse attention methods exploit the inherent sparsity in attention to speed up the prefilling phase of long-context inference, mitigating the quadratic complexity of full attention computation. While existing sparse attention methods rely on predefined patterns or inaccurate estimations to approximate attention behavior, they often fail to fully capture the true dynamics of attention, resulting in reduced efficiency and compromised accuracy. Instead, we propose a highly accurate sparse attention mechanism that shares similar yet precise attention patterns across heads, enabling a more realistic capture of the dynamic behavior of attention. Our approach is grounded in two key observations: (1) attention patterns demonstrate strong inter-head similarity, and (2) this similarity remains remarkably consistent across diverse inputs. By strategically sharing computed accurate patterns across attention heads, our method effectively captures actual patterns while requiring full attention computation for only a small subset of heads. Comprehensive evaluations demonstrate that our approach achieves superior or comparable speedup relative to state-of-the-art methods while delivering the best overall accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。