让压缩键值缓存生成结果与全量缓存一致,同时保持高速推理。
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
- 用压缩缓存预生成文本,再用完整缓存验证,确保输出一致。
- 实测吞吐最高达全量缓存的4倍,且输出完全相同。
- 适用于长上下文和远程前缀缓存,兼容多种压缩方法。
随着上下文长度增加,键值(KV)缓存的庞大尺寸已成为大语言模型服务的主要瓶颈。为此,许多压缩方法如丢弃令牌和量化被提出,但几乎都存在固有损耗——尽管短输出时精度下降微小,但随着解码进行,输出会逐渐偏离全量缓存结果,导致代码生成和工具调用出现灾难性失败。本文提出VeriCache,首个能保证与全量缓存解码输出完全一致,同时大幅保留多种压缩算法高吞吐优势的推理框架。VeriCache利用压缩缓存起草文本,再与全量缓存对比验证。关键系统挑战在于:将全量缓存保留在显存外,并最小化其调入验证的开销。核心洞察为:压缩解码与全量缓存交换可并行,因前者受显存带宽限制,后者受PCIe/网络限制;且压缩缓存常产生接近全量的结果,允许长稿写周期分摊每次全量缓存调入代价。VeriCache适用于长上下文解码和远程前缀缓存,通过统一压缩器接口支持广泛令牌丢弃与量化方法,并可与传统推测解码协同。实验表明,相比全量缓存推理,VeriCache吞吐最高提升4倍,且输出完全一致。
原文摘要 · Abstract (English)
The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token dropping and quantization, have been proposed. However, almost all of these methods are inherently lossy-despite minimal accuracy degradation for short outputs, their outputs increasingly diverge from full-KV-cache outputs as more tokens are decoded, which leads to catastrophic failures in code generation and tool calling. We present VeriCache, the first inference framework that ensures the same output as full-KV-cache decoding but largely preserves the high decoding throughput of a range of KV cache compression algorithms. VeriCache uses the compressed KV cache to draft tokens, then verifies them against the full KV cache. While it may seem like just speculative decoding, VeriCache requires addressing a key system challenge to work-keeping the full KV cache out of GPU memory and minimizing the overhead of swapping it in for verification. The insight is two-fold: (1) compressed-KV decoding can be parallelized with full-KV swap, because one is HBM-bandwidth-bound and the other is PCIe/network-bound, and (2) the compressed KV cache often produces output similar to the full KV cache, allowing a long drafting horizon to amortize each full-KV swap. VeriCache applies to both long-context decoding and remote prefix caching, supports a broad family of token-dropping and quantization methods through a uniform compressor interface, and composes with traditional speculative decoding. Experimental results show that VeriCache achieves up to 4X higher throughput than full-KV inference while producing identical outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。