通过重建上下文压缩KV缓存，实现跨查询高效复用。
KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
- 用模型自重建原始上下文来评估KV对重要性，决定是否淘汰。
- 缓存大小缩小3-4倍，解码延迟降低约2倍，性能损失极小。
- 适合长上下文推理场景，尤其在多查询任务中表现更优。
基于Transformer的大语言模型在推理时会缓存上下文的键值（KV）对。随着上下文长度增加，KV缓存规模急剧扩大，导致显著内存开销和注意力计算延迟。本文提出KVzip，一种与查询无关的KV缓存淘汰方法，支持压缩后缓存在不同查询间的有效复用。KVzip利用底层大模型从缓存的KV对中重建原始上下文，量化每个KV对的重要性，并淘汰重要性较低的条目。大量实证评估表明，KVzip可使KV缓存规模缩小3至4倍，FlashAttention解码延迟降低约2倍，同时在问答、检索、推理和代码理解等任务中性能损失微乎其微。实验涵盖LLaMA3.1、Qwen2.5、Gemma3等多种模型，上下文长度最高达17万词元。相比现有依赖查询的缓存淘汰方法，KVzip在多查询场景下即使缓存预算占比达90%仍保持稳定性能，显著优于同类方法。
原文摘要 · Abstract (English)
Transformer-based large language models (LLMs) cache context as key-value (KV) pairs during inference. As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency. This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries. KVzip quantifies the importance of a KV pair using the underlying LLM to reconstruct original contexts from cached KV pairs, subsequently evicting pairs with lower importance. Extensive empirical evaluations demonstrate that KVzip reduces KV cache size by $3$-$4\times$ and FlashAttention decoding latency by approximately $2\times$, with negligible performance loss in question-answering, retrieval, reasoning, and code comprehension tasks. Evaluations include various models such as LLaMA3.1, Qwen2.5, and Gemma3, with context lengths reaching up to 170K tokens. KVzip significantly outperforms existing query-aware KV eviction methods, which suffer from performance degradation even at a 90% cache budget ratio under multi-query scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。