通过相似性重建实现灵活压缩，显著降低大模型长文本推理内存占用。
EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
- 利用注意力头间与层内相似性，用轻量网络从部分缓存重建丢弃内容。
- 在多个压缩比下优于现有方法，短上下文仍保持全缓存吞吐性能。
- 仅需单卡几分钟微调，支持随时切换压缩与完整缓存模式。
随着大语言模型在长上下文应用中对键值（KV）缓存内存需求的增长，内存瓶颈日益突出。现有低秩压缩方法通过修改模型投影来减小缓存体积，但限制了在内存充足时切换回标准全缓存推理的灵活性。本文提出EchoKV，一种支持按需在全缓存与压缩缓存间切换的灵活压缩框架。不同于传统压缩-解压范式，EchoKV采用轻量级网络，从部分子集重建被丢弃的KV组件，利用注意力头间的内在层间与层内相似性。我们进一步设计了一种轻量级两阶段微调策略，仅需在单张A100 GPU上数分钟即可完成7B模型的训练。在LongBench和RULER数据集上的实验表明，EchoKV在多种压缩比和骨干模型下均持续优于现有方法，且在短上下文场景中保持全缓存推理的吞吐性能。
原文摘要 · Abstract (English)
The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。