提出全局资源分配方法,让每块缓存都精准匹配上下文需求。
Every Cache Entry Earns Its Place: Global Allocation of Resolution and Coverage for KV Cache Compression

- 将缓存压缩建模为全局资源分配问题,按层-头-槽原子单元优化
- 在128倍压缩下仍保持性能稳定,32项测试中24项排名第一
- 无需训练,纯GPU运行,适合长文本生成与推理场景
随着大语言模型处理更长上下文,键值缓存(KV cache)的存储与重复访问已成为主要瓶颈。现有压缩方法依赖预设固定规则,通常围绕令牌淘汰或合并设计,导致缓存资源无法在层、头和上下文槽之间自由流动,也无法协同分配以平衡局部分辨率与信息覆盖率。为此,我们提出GraceKV,一种面向KV缓存压缩中分辨率与覆盖范围的全局分配方法,将压缩过程形式化为固定缓存预算下的全局资源分配问题。GraceKV将每一层-KV头-槽组合视为原子单元,构建原型树:叶节点对应令牌级KV条目,内部节点用单一原型压缩其子节点覆盖的KV空间。树中一组不重叠节点构成原子单元的表示;新增树根可扩展信息覆盖,分裂选定节点则提升局部分辨率。所有候选操作在全球共享缓存预算下竞争。最终,各树保留的节点构成压缩后的KV缓存。该过程自适应地全局决定缓存资源在原子单元间的分配及分辨率与覆盖率的平衡。GraceKV无需额外训练,整个压缩与推理过程在GPU上完成。跨多种长上下文任务与压缩比的系统实验表明,GraceKV在32项设置中24项排名第一,且在128倍压缩下仍具鲁棒性。结果验证了全局预算分配在协调信息覆盖与局部分辨率方面的有效性。
原文摘要 · Abstract (English)
As large language models (LLMs) process increasingly long contexts, KV cache storage and repeated access have become a major bottleneck. Existing KV cache compression methods rely on predefined, fixed compression rules and are typically developed around either token eviction or merging. As a result, cache resources can neither flow freely across layers, heads, and context slots, nor be jointly allocated to balance local resolution and information coverage. Therefore, we propose GraceKV, a global approach for the allocation of resolution and coverage in KV cache compression, and formulate the compression process as a global resource allocation problem under a fixed cache budget. GraceKV treats each layer-KV head-slot combination as an atomic unit and builds a prototype tree. Leaf nodes correspond to token-level KV entries, while each internal node uses a single prototype to compress the KV space covered by its children. A set of non-overlapping nodes in the tree forms the representation of an atomic unit. Adding the root of a new tree expands information coverage, whereas splitting a selected node improves local resolution. All candidate actions compete globally for a shared cache budget. Finally, the nodes retained across all trees form the compressed KV cache. This process adaptively determines the allocation of cache resources among atomic units globally and the balance between resolution and coverage. GraceKV requires no additional training, and the entire compression and inference process is performed on the GPU. Systematic experiments across diverse long-context tasks and compression ratios show that GraceKV ranks first in 24 of 32 settings and remains robust up to 128-fold compression. These results validate the effectiveness of global budget allocation in coordinating information coverage and local resolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。