提出可逆压缩KV缓存技术，解决长文本推理内存瓶颈
KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction
- 用Sketch算法构建可重建的压缩结构，避免信息永久丢失
- 2k上下文仅需10%缓存，32k上下文仅需25%缓存，精度损失<2%
- 适合需要长上下文、高内存效率的LLM部署场景
随着大语言模型上下文长度迅速增加，键值（KV）缓存的内存需求已成为模型部署和批处理的瓶颈。传统压缩方法通常永久丢弃或不可逆合并低注意力分数的‘不重要’令牌，导致无法恢复的信息丢失，我们称之为‘上下文遗忘’，显著降低模型的信息检索能力。为此，我们提出基于Sketch算法的可逆KV缓存压缩方法KVReviver。该方法通过额外数据结构重建压缩令牌，实现有限内存下的完整计算。实验表明，在2048长度上下文中，仅需10%的KV缓存预算即可保持相同的端到端推理精度；在32768长度上下文中，仅需25%的预算即可达到相当或相近的精度（约2%精度损失）。
原文摘要 · Abstract (English)
As the context length of current large language models (LLMs) rapidly increases, the memory demand for the Key-Value (KV) cache is becoming a bottleneck for LLM deployment and batch processing. Traditional KV cache compression methods typically involve permanently evicting or irreversibly merging "less important" tokens with low attention scores. This approach results in the unrecoverable loss of token information, which we call Contextual Amnesia, significantly degrading the model's information retrieval capability. To address this issue, we propose KVReviver, a reversible KV cache compression method based on the sketch algorithm. This method allows reconstructing compressed tokens from an additional data structure, thus enabling full-scale computation within limited memory. Experiments showed that in 2k-length contexts, it requires only 10% of KV Cache budget while maintaining identical end-to-end inference accuracy. For 32k-length contexts, it achieves equivalent or comparable accuracy ~2% accuracy loss) using merely 25% of KV Cache budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。