用压缩技术让大模型推理时的缓存占用减少20倍，节省显存。
KV Cache Transform Coding for Compact Storage in LLM Inference
- 基于主成分分析与自适应量化，对键值缓存进行轻量级压缩
- 最高达40倍压缩比，长文本和推理任务准确率几乎不变
- 无需改动模型，适合部署在显存有限的推理系统中
大规模语言模型服务需要高效的键值（KV）缓存管理。通过共享前缀提示，对话中的多个轮次可复用缓存，但过期缓存会占用稀缺的GPU内存，导致需要换出或重新计算。本文提出一种轻量级变换编码器KVTC，用于高效压缩KV缓存，实现紧凑的显卡内与显卡外存储。受传统媒体压缩启发，KVTC结合了基于PCA的特征去相关、自适应量化和熵编码，仅需短暂初始校准，且不改变模型参数。通过利用KV缓存中的冗余信息，该方法在保持推理与长上下文准确性的同时，实现了最高20倍的压缩比，特定场景下可达40倍及以上。我们在Llama 3、Mistral NeMo和R1-Qwen 2.5模型上，针对AIME25、GSM8K、LiveCodeBench、LongBench、MATH-500、MMLU、Qasper和RULER等基准进行了测试，结果表明，其性能优于现有的推理时基线方法（如令牌淘汰、量化、SVD方法），同时具备更高的压缩率。这表明KVTC是构建可复用缓存、实现高效内存管理的大模型服务的重要组件。
原文摘要 · Abstract (English)
Serving large language models (LLMs) at scale necessitates efficient key-value (KV) cache management. KV caches can be reused across conversation turns via shared-prefix prompts that are common in iterative code editing and chat. However, stale caches consume scarce GPU memory, require offloading, or force recomputation. We present KVTC, a lightweight transform coder that compresses KV caches for compact on-GPU and off-GPU storage. Drawing on classical media compression, KVTC combines PCA-based feature decorrelation, adaptive quantization, and entropy coding. It requires only a brief initial calibration and leaves model parameters unchanged. By exploiting redundancies in KV caches, KVTC achieves up to 20$\times$ compression while maintaining reasoning and long-context accuracy, and 40$\times$ or higher for specific use cases. We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks including AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER. It consistently outperforms inference-time baselines such as token eviction, quantization, and SVD-based methods, while achieving higher compression ratios. These results support KVTC as a practical building block for memory-efficient LLM serving with reusable KV caches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。