KVzap通过自适应剪枝实现快速推理，大幅压缩缓存且几乎不损失精度。
KVzap: Fast, Adaptive, and Faithful KV Cache Pruning
- 基于输入动态调整剪枝策略，兼顾预填充与解码阶段效率。
- 在长文本和推理任务中实现2到4倍的缓存压缩，精度几乎无损。
- 适合追求高速推理的模型部署者，尤其适用于大上下文场景。
Transformer语言模型的上下文长度持续增长，导致键值（KV）缓存成为关键的推理瓶颈。尽管已有多种KV缓存剪枝方法提出，但因速度与准确率之间的权衡，尚未被主流推理引擎采纳。我们提出KVzap，一种快速、输入自适应的KVzip近似方法，适用于预填充和解码阶段。在Qwen3-8B、Llama-3.1-8B-Instruct和Qwen3-32B上，针对长上下文和推理任务，KVzap实现了2至4倍的KV缓存压缩，精度损失可忽略，并在KVpress排行榜上达到最先进水平。代码与模型已公开于https://github.com/NVIDIA/kvpress。
原文摘要 · Abstract (English)
Growing context lengths in transformer-based language models have made the key-value (KV) cache a critical inference bottleneck. While many KV cache pruning methods have been proposed, they have not yet been adopted in major inference engines due to speed--accuracy trade-offs. We introduce KVzap, a fast, input-adaptive approximation of KVzip that works in both prefilling and decoding. On Qwen3-8B, Llama-3.1-8B-Instruct, and Qwen3-32B across long-context and reasoning tasks, KVzap achieves $2$--$4\times$ KV cache compression with negligible accuracy loss and achieves state-of-the-art performance on the KVpress leaderboard. Code and models are available at https://github.com/NVIDIA/kvpress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。