通过语义匹配复用相似提示的缓存，显著提升大模型推理速度。
SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching
- 用局部敏感哈希匹配词元语义，实现非字面相似提示的缓存共享
- 5000词输入下提速6.25倍，显存降低42%，质量几乎无损
- 适合多文档摘要、对话系统等语义相近但表达不同的场景
随着大语言模型持续扩展，推理过程中的键值（KV）缓存内存开销已成为主要瓶颈。现有方法多聚焦于单个提示内的缓存压缩，或复用跨提示的公共前缀与高频文本片段，但在提示语义相似但词汇不同（如多文档摘要、对话系统）时效果有限。本文提出SemShareKV，一种基于词元级局部敏感哈希（LSH）匹配的语义感知缓存共享与压缩框架，通过在词元嵌入上应用模糊匹配并结合旋转位置编码（RoPE）以更好保留位置信息，选择性复用参考提示的键值对。实验表明，在多个摘要数据集上，5000词输入时可实现最高6.25倍加速，显存占用降低42%，且输出质量几乎无损。该结果凸显了语义感知缓存共享在高效大模型推理中的潜力。
原文摘要 · Abstract (English)
As large language models (LLMs) continue to scale, the memory footprint of key-value (KV) caches during inference has become a significant bottleneck. Existing approaches primarily focus on compressing KV caches within a single prompt or reusing shared prefixes or frequently ocurred text segments across prompts. However, such strategies are limited in scenarios where prompts are semantically similar but lexically different, which frequently occurs in tasks such as multi-document summarization and conversational agents. We propose \textit{SemShareKV}, a KV cache sharing and compression framework that accelerates LLM inference by reusing KVCache in semantically similar prompts. Instead of relying on exact token matches, SemShareKV applies fuzzy token matching using locality-sensitive hashing (LSH) on token embeddings and incorporates Rotary Position Embedding (RoPE) to better preserve positional information. By selectively reusing relevant key-value pairs from a reference prompt's cache, SemShareKV reduces redundant computation while maintaining output quality. Experiments on diverse summarization datasets show up to 6.25$\times$ speedup and 42\% lower GPU memory usage with 5k tokens input, with negligible quality degradation. These results highlight the potential of semantic-aware cache sharing for efficient LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。