用轻量门控机制高效压缩大模型缓存，70%缓存可删性能几乎不变。
Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
- 通过门控模块识别关键键值对，只保留重要信息
- 最高可删除70%缓存，推理性能几乎无损
- 无需反向传播训练，适合各类大模型部署
高效的关键-值（KV）缓存管理对大语言模型（LLMs）的实际部署至关重要，但现有压缩技术常在性能下降与计算开销之间权衡。本文提出一种针对固定权重LLM的新型门控式KV缓存淘汰方法，可在极低计算成本下实现高压缩比。该方法引入轻量级的sink-attention门控模块，识别并保留关键的KV对，并可无缝集成至预填充和解码阶段。门控训练仅依赖LLM的前向传播，避免昂贵的反向传播，同时通过任务无关的重建目标实现强任务泛化能力。在Qwen2.5-1M、Qwen3和Gemma3系列模型上的大量实验表明，该方法在删除高达70%的KV缓存时仍保持近乎无损的性能，且在长文本理解、代码理解及数学推理等多样化任务中表现一致稳定，验证了其普适性。
原文摘要 · Abstract (English)
Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。