arXiv:2605.07234cs.CLcs.AI2026-05

提出新方法,用全局评分压缩长文本推理的缓存,性能更稳

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference

论文配图:Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
图 1 · 摘自论文原文
  • 将缓存淘汰重构为输出感知的矩阵近似问题,融合多头交互
  • 仅用5%缓存仍保持性能,极端压缩下准确率损失减半
  • 首次实现全模型统一重要性评分,适合长文本生成场景

大语言模型在长文本推理中面临键值缓存(KV Cache)增长带来的内存与运行时开销。现有淘汰策略主要依赖局部注意力权重,忽视了值表示、输出投影和多头间交互的影响。本文将缓存淘汰重新定义为输出感知的层级矩阵乘法近似问题,提出LaProx新策略,显式建模注意力图与投影值状态间的乘积交互,精准量化每个词元贡献,并考虑多头依赖关系。基于此度量,我们设计首个全局可比的重要度评分机制,实现全模型统一选择而非局部头级决策。在LongBench和Needle-In-A-Haystack共19个数据集上的实验表明,本方法仅需5%的KV缓存即可维持模型性能,且在所有配置下均优于现有方法;尤其在极端压缩下,相比当前最优基线,准确率损失降低达2倍,额外开销极小。

原文摘要 · Abstract (English)

Large language models (LLMs) support long-context inference but suffer from substantial memory and runtime overhead due to Key-Value (KV) Cache growth. Existing KV Cache eviction methods primarily rely on local attention weights, neglecting the influence of value representations, output projection, and inter-head interactions. In this work, we reformulate KV Cache eviction from a conventional head-wise, weight-averaging approach into an output-aware, layer-wise matrix multiplication approximation problem. We introduce LaProx, a novel eviction strategy that explicitly models the multiplicative interaction between attention maps and projected value states to accurately quantify token contributions while accounting for inter-head dependencies. Building on this metric, we propose the first unified eviction strategy that assigns globally comparable importance scores to tokens, enabling model-wide selection instead of local, head-wise decisions. Experimental results across 19 datasets on long-context benchmarks LongBench and Needle-In-A-Haystack demonstrate that our approach maintains model performance with only 5\% of the KV cache and consistently outperforms prior works across all configurations. Notably, our method achieves up to 2$\times$ accuracy loss reduction under extreme compression scenarios compared to existing state-of-the-art baselines with minimal overhead.

长文本推理缓存优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。