提出自适应注意力稀疏框架，动态优化长文本生成效率。
Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning
- 引入分层top-p剪枝，实现注意力稀疏的自适应预算分配。
- 可自动剪除最多98%冗余词元，推理速度提升15.4倍。
- 兼容现有稀疏注意力方法，适合部署长上下文大模型场景。
利用注意力稀疏性加速长上下文大语言模型（LLMs）已成为热门研究方向。然而，当前如稀疏注意力或键值（KV）缓存压缩等算法通常采用固定预算，在实际部署中面临挑战，因其无法适应真实场景中准确率与效率之间动态变化的最优平衡。本文发现，将top-p采样（nucleus sampling）引入注意力稀疏化，可意外实现自适应预算控制。基于此，我们提出Twilight框架，无需牺牲原有稀疏注意力算法的准确性，即可为任意现有方法引入自适应稀疏性。实验表明，Twilight可自适应剪除最多98%冗余词元，在长上下文LLM解码中实现自注意力操作15.4倍加速，端到端每词元延迟提升3.9倍。
原文摘要 · Abstract (English)
Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-value (KV) cache compression tend to use a fixed budget, which presents a significant challenge during deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. In this paper, we find that borrowing top-$p$ sampling (nucleus sampling) to sparse attention can surprisingly achieve adaptive budgeting. Based on this, we propose Twilight, a framework to bring adaptive sparsity to any existing sparse attention algorithm without sacrificing their accuracy. Empirical results show that Twilight can adaptively prune at most 98% of redundant tokens, leading to $15.4\times$ acceleration in self-attention operations and $3.9\times$ acceleration in end-to-end per token latency in long context LLM decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。