通过语义分块与聚类压缩注意力缓存，显著提速且不丢性能。
SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging
- 按语言语义边界分块，用贪心聚类算法合并相似语义片段。
- 在保持模型性能前提下，推理速度最高提升2.61倍，内存减少明显。
- 适合需要高效长文本生成的场景，如对话系统、内容创作。
现有KV缓存压缩方法多基于离散词元或非语义分块，常导致语义碎片化，造成不可逆信息损失并降低模型性能。为此，我们提出SemantiCache，一种以语言语义层次结构为指导的新型压缩框架。首先，通过分隔符将缓存划分为语义连贯的区块；在每个区块内，引入计算高效的贪心种子聚类（GSC）算法，将词元聚合成语义簇；这些簇进一步合并为语义核心，并通过比例注意力机制重新平衡被压缩词元的注意力贡献。在多个基准和模型上的实验表明，SemantiCache可使推理阶段解码速度最高提升2.61倍，大幅降低内存占用，同时保持与原始模型相当的性能。
原文摘要 · Abstract (English)
Existing KV cache compression methods generally operate on discrete tokens or non-semantic chunks. However, such approaches often lead to semantic fragmentation, where linguistically coherent units are disrupted, causing irreversible information loss and degradation in model performance. To address this, we introduce SemantiCache, a novel compression framework that preserves semantic integrity by aligning the compression process with the semantic hierarchical nature of language. Specifically, we first partition the cache into semantically coherent chunks by delimiters, which are natural semantic boundaries. Within each chunk, we introduce a computationally efficient Greedy Seed-Based Clustering (GSC) algorithm to group tokens into semantic clusters. These clusters are further merged into semantic cores, enhanced by a Proportional Attention mechanism that rebalances the reduced attention contributions of the merged tokens. Extensive experiments across diverse benchmarks and models demonstrate that SemantiCache accelerates the decoding stage of inference by up to 2.61 times and substantially reduces memory footprint, while maintaining performance comparable to the original model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。