提出无需预设阈值的KV缓存压缩方法，提升大模型推理稳定性。
ReFreeKV: Towards Threshold-Free KV Cache Compression
- 设计自适应预算分配机制，摆脱对输入敏感阈值的依赖。
- 在13个数据集上实现与完整缓存相当的性能，内存消耗显著降低。
- 适合需要泛化能力的开放域大模型推理场景。
为降低大模型推理时的内存占用，已有若干方法用于KV缓存剪枝。尽管这些技术在多数数据集上可实现无损内存压缩，但通常依赖一个被忽视的条件：需预先设定针对特定输入/领域的阈值以获得最佳性能。然而，现实场景中开放域输入涵盖多样领域、长度和难度，缺乏明确的阈值划分边界，导致此类依赖阈值的设计存在根本性局限，可能在任意输入下引发性能大幅下降。为此，本文提出一种新目标，推动无阈值的鲁棒KV压缩，倡导自适应预算分配以保持全缓存性能。我们进一步提出首个实现该目标的方法——ReFreeKV。在包含13个数据集、覆盖多种上下文长度、任务类型及模型规模的实验中，充分验证了其有效性与高效性。代码已开源：https://github.com/Patrick-Ni/ReFreeKV。
原文摘要 · Abstract (English)
To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning. While these techniques can accomplish lossless memory reduction on many datasets, they often hinge on an under-emphasized condition: an input/domain-specific threshold for KV cache budget needs to be pre-determined to achieve the optimal performance. However, such input-sensitive design may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for threshold selection. As a result, the dependence of such input-sensitive threshold can be a fundamental limitation that causes large degradation on arbitrary inputs. In this work, we propose a new objective that lifts the threshold constraints for robust KV compression, advocating for "threshold-free" methods that adaptively adjust budget allocation while preserving full-cache performance. We then propose a novel method, ReFreeKV, serving as the first instantiation of this objective. Extensive experiments across 13 datasets with diverse context lengths, task types, and model sizes demonstrate its efficacy and efficiency. Our code is publicly released at https://github.com/Patrick-Ni/ReFreeKV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。