提出SpindleKV方法，平衡深浅层缓存压缩，提升推理效率。
SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers
- 深层用注意力权重淘汰，浅层用相似性学习的码本替换
- 在三个模型上实现更高缓存压缩率，性能不降反升
- 适合部署大模型的低显存设备，尤其优化了组查询注意力
大语言模型近年来取得显著进展，但KV缓存带来的内存消耗已成为推理系统的重大挑战。已有淘汰策略揭示了缓存中存在固有冗余，尤其在深层表现突出，但浅层压缩效果不足。基于观察到的缓存高度相似性，我们提出新型缓存压缩方法SpindleKV，兼顾浅层与深层优化。深层采用基于注意力权重的淘汰机制，浅层则引入由相似性与合并策略训练的码本替换方案。此外，SpindleKV有效解决了其他基于注意力淘汰方法在组查询注意力（GQA）场景下的困境。在两个通用基准和三种不同大模型上的实验表明，相比基线方法，SpindleKV实现了更优的缓存压缩效果，同时保持甚至提升模型性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved impressive accomplishments in recent years. However, the increasing memory consumption of KV cache has possessed a significant challenge to the inference system. Eviction methods have revealed the inherent redundancy within the KV cache, demonstrating its potential for reduction, particularly in deeper layers. However, KV cache reduction for shallower layers has been found to be insufficient. Based on our observation that, the KV cache exhibits a high degree of similarity. Based on this observation, we proposed a novel KV cache reduction method, SpindleKV, which balances both shallow and deep layers. For deep layers, we employ an attention weight based eviction method, while for shallow layers, we apply a codebook based replacement approach which is learnt by similarity and merging policy. Moreover, SpindleKV addressed the Grouped-Query Attention (GQA) dilemma faced by other attention based eviction methods. Experiments on two common benchmarks with three different LLMs shown that SpindleKV obtained better KV cache reduction effect compared to baseline methods, while preserving similar or even better model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。