自动压缩大模型的键值缓存，省去手动设预算，内存减半且准确率不降。
Adaptive KV-Cache Compression without Manually Setting Budget
- 通过采样未来查询预测关键信息，动态决定缓存大小。
- 在多个基准上实现2倍内存减少，准确率保持或提升。
- 适合需要高效推理且不想调参的场景。
大型语言模型推理严重依赖键值缓存（KV-cache）加速自回归解码，但缓存内存随序列长度迅速增长，带来显著效率挑战。现有压缩方法存在‘普罗克拉斯提斯之床’问题：强制不同负载采用固定压缩比，导致资源分配和推理性能不佳。为此，我们提出GVote，一种无需手动设定预算的自适应KV缓存压缩方案，实现了更优的准确性-效率权衡。GVote基于重要键是未来查询所需键的聚合这一原则，通过蒙特卡洛式采样潜在查询并聚合选定键，来无须人工指定地确定最优缓存预算。实验表明，GVote在GSM8K、RULER和Longbench等多个基准上均有效。相比基线方法，其内存减少2倍，同时准确率保持更高或相当。
原文摘要 · Abstract (English)
Large language models (LLMs) inference relies heavily on KV-caches to accelerate autoregressive decoding, but the resulting memory footprint grows rapidly with sequence length, posing significant efficiency challenges. Current KV-cache compression methods suffer from a Procrustes' bed problem: they force diverse workloads into fixed compression ratios, leading to suboptimal resource allocation and inference performance. To this end, we present GVote, an adaptive KV-cache compression scheme that eliminates manual budget specification while achieving superior accuracy-efficiency trade-offs. GVote operates on the principle that the important keys are the aggregation of keys required by future queries. The method predicts future query attention demands by Monte-Carlo style sampling potential queries and aggregating selected keys to determine the optimal cache budget without manual specification. Experimental evaluation demonstrates GVote's effectiveness across multiple benchmarks, including GSM8K, RULER and Longbench. Compared to baselines, GVote exhibits 2$\times$ memory reduction while the accuracy maintains higher or comparable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。