通过共享不相似的键值缓存，显著降低大模型推理内存占用
KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
- 跨层共享不相似的键值缓存，反直觉提升压缩效率
- 减少30%计算量，内存下降且生成速度提升1.3倍以上
- 兼容现有压缩方法，适合部署大模型的工程优化
大型语言模型的发展导致模型规模急剧膨胀，推理阶段对GPU内存需求巨大。注意力机制中的键值缓存（KV cache）占用超过80%的内存。当前多数缓存压缩方法聚焦于单个Transformer层内的压缩，而较少关注跨层压缩。本文提出一种即插即用的方法KVSharer，通过层间共享实现跨层压缩。令人意外的是，我们发现共享不相似的KV缓存反而更有利于保持模型性能。实验表明，该方法可降低30%的KV缓存计算量，有效减少内存消耗，同时生成速度提升至少1.3倍。此外，验证了其与现有层内压缩方法的兼容性，两者结合可进一步节省内存。
原文摘要 · Abstract (English)
The development of large language models (LLMs) has significantly expanded model sizes, resulting in substantial GPU memory requirements during inference. The key and value storage of the attention map in the KV (key-value) cache accounts for more than 80\% of this memory consumption. Nowadays, most existing KV cache compression methods focus on intra-layer compression within a single Transformer layer but few works consider layer-wise compression. In this paper, we propose a plug-and-play method called \textit{KVSharer}, which shares the KV cache between layers to achieve layer-wise compression. Rather than intuitively sharing based on higher similarity, we discover a counterintuitive phenomenon: sharing dissimilar KV caches better preserves the model performance. Experiments show that \textit{KVSharer} can reduce KV cache computation by 30\%, thereby lowering memory consumption without significantly impacting model performance and it can also achieve at least 1.3 times generation acceleration. Additionally, we verify that \textit{KVSharer} is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。