通过融合多个请求的KV缓存块，显著降低内存占用并提升大模型并发推理速度。
Joint Encoding of KV-Cache Blocks for Scalable LLM Serving
- 将不同请求的相似缓存块合并为共享表示，保持原有结构不变。
- 最高实现4.38倍缓存压缩，准确率损失极小，支持多模型通用。
- 无需专用硬件，在单机上提升约40%生成吞吐量，适合实际部署。
现代大语言模型驱动交互式AI系统，但键值（KV）缓存的内存开销限制了高并发场景下的实时吞吐量。现有压缩方法依赖固定规则，破坏张量布局或需特殊计算，难以扩展和部署。本文提出联合编码KV缓存块：将跨请求与输入片段的相似块融合为共享表示，同时保持标准缓存结构。该方法缓解了缓存内存瓶颈，实现无专用硬件的高并发服务。理论上，基于泊松过程分析融合块的率失真权衡；实验上，本方法在多种大模型与基准测试中实现最高4.38倍的缓存压缩，精度损失可忽略，优于近期结构化与自适应压缩基线。在真实vLLM单机部署中，推断吞吐量提升约40%，显著改善推理效率。代码已开源。
原文摘要 · Abstract (English)
Modern large language models (LLMs) drive interactive AI systems but are bottlenecked by the memory-heavy growth of key-value (KV) caches, which limits real-time throughput under concurrent loads. Existing KV-cache compression methods rely on rigid heuristics, disrupt tensor layouts, or require specialized compute, hindering scalability and deployment. We propose joint encoding of KV-cache blocks, which fuses similar blocks across requests and input chunks into shared representations while preserving standard cache structure. This alleviates the KV-cache memory bottleneck, supporting high-concurrency serving without specialized hardware. Theoretically, we analyze the rate-distortion tradeoff of fused cache blocks under a Poisson process model. Empirically, our method achieves up to 4.38 $\times$ KV-cache compression with negligible accuracy loss across diverse LLMs and benchmarks, outperforming recent structured and adaptive compression baselines. In real LLM serving, joint encoding improves the token throughput by $\sim$40\% on a single-machine vLLM benchmark, demonstrating substantial gains in inference throughput. Code is available at https://github.com/sef1/kv_fast_fusion kv_joint_encoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。