提出无需训练的KV缓存压缩方法,显著减少长文本生成内存占用
GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs

- 用岭回归实现全局信息分配,避免合并集中于边界令牌
- 在LongBench和RULER上保持性能不降,仅需极低额外开销
- 适合需要长上下文且资源受限的部署场景
长上下文大语言模型依赖键值(KV)缓存支持注意力机制。但维护完整缓存带来巨大内存开销,促使研究者采用固定预算的淘汰与合并策略。现代淘汰方法趋向保留连续片段,因其更利于语义连贯性。然而,与淘汰后合并结合时,该策略使合并集中在少数片段边界载体令牌上,造成严重失衡,加剧过度合并与信息丢失。为此,我们提出GRKV(全局回归用于KV缓存),一种无需训练的合并方法,直接最小化压缩缓存与完整缓存注意力输出之间的差异。GRKV通过基于岭回归的合并步骤,将被移除令牌的信息分布至保留令牌,同时正则化更新以防止过平滑。在LongBench与RULER长上下文基准测试中,GRKV是唯一能在极小开销下提升整体性能的合并方法。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) with extended context lengths rely on the key-value (KV) cache to support attention over prior tokens. However, maintaining the KV cache incurs substantial memory overhead, motivating KV-cache compression methods that enforce a fixed budget through eviction and merging. Modern eviction methods increasingly adopt span-based retention because preserving contiguous spans is empirically effective and better preserves semantic coherence. Yet, when combined with post-eviction merging, span-based retention concentrates merges onto a small set of span-boundary carrier tokens, producing a highly imbalanced merge pattern that exacerbates over-merging and increases information loss. To address this imbalance, we propose GRKV (Global Regression for KV Cache), a training-free KV-cache merging method that directly minimizes the discrepancy between compressed-cache and full-cache attention outputs. GRKV uses ridge-regression-based merge steps to distribute information from evicted tokens across retained tokens, while regularizing the updates to prevent over-smoothing. Across the LongBench and RULER long-context benchmarks, GRKV is the only merging method that improves overall performance with minimal overhead. Our code is available at https://github.com/pjunjie/GRKV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。