通过预测未来查询分布来压缩注意力缓存，无需训练即可高效推理。
Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
- 用未来查询分布估算关键值对的重要性，实现无训练压缩。
- 在预填充和解码阶段均优于现有方法，性能无损失。
- 适合需要高效大模型推理的研究者与开发者使用。
大型语言模型推理中的键值（KV）缓存内存消耗是主要瓶颈。尽管基于注意力分数的KV缓存剪枝有潜力，但面临实际限制：压缩时无法获取未来标记的注意力分数，且现代实现如Flash Attention不生成完整的注意力矩阵，导致过去分数不可访问。为此，我们提出一种无需训练的压缩方法——预期注意力（Expected Attention），通过预测未来查询对各KV对的关注程度，估计其重要性。该方法利用大模型激活的分布特性，对每个KV对闭式计算预期注意力分数，实现有原则的排序与剪枝，对残差流影响极小，有效压缩缓存且无性能下降。该方法可无缝应用于预填充和解码阶段，在两种场景下均持续超越当前最优基线。最后，我们发布了KVPress——一个包含20余种压缩技术的综合性库，助力研究者实现与基准测试。
原文摘要 · Abstract (English)
Memory consumption of the Key-Value (KV) cache represents a major bottleneck for efficient large language model inference. While attention-score-based KV cache pruning shows promise, it faces critical practical limitations: attention scores from future tokens are unavailable during compression, and modern implementations like Flash Attention do not materialize the full attention matrix, making past scores inaccessible. To overcome these challenges, we introduce $\textbf{Expected Attention, a training-free compression method}$ that estimates KV pairs importance by predicting how future queries will attend to them. Our approach leverages the distributional properties of LLM activations to compute expected attention scores in closed form for each KV pair. These scores enable principled ranking and pruning of KV pairs with minimal impact on the residual stream, achieving effective compression without performance degradation. Importantly, our method operates seamlessly across both prefilling and decoding phases, consistently outperforming state-of-the-art baselines in both scenarios. Finally, $\textbf{we release KVPress, a comprehensive library to enable researchers to implement and benchmark KV cache compression methods, already including more than 20 techniques}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。