无需训练，通过自适应量化与均值中心化压缩注意力缓存，大幅降低内存占用。
TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
- 自适应调整各层量化精度，匹配误差敏感度
- 均值中心化消除异常值处理需求，内存降至16位基线的27%
- 适用于长上下文、长思维链推理场景
Transformer模型中的键值（KV）缓存是高效解码的关键组件，但其内存开销随序列长度增长而急剧上升，制约了大语言模型的可扩展部署。在多种KV缓存压缩方法中，键值激活的量化被广泛研究。然而，多数量化方法仍需单独处理稀疏且不连续的异常值。为此，我们提出TaDA，一种无需训练的KV缓存压缩方案，通过量化精度自适应于各层的误差敏感性，并引入均值中心化以消除对异常值的单独处理。该方法在多个支持不同上下文长度的模型上实现显著精度提升。实验表明，在标准基准测试中，该技术将KV缓存内存占用降至原始16位基线的27%，同时保持相近准确率。本方法为长上下文语言模型推理、复杂推理任务和长思维链提供了高可扩展性与高性能支持。
原文摘要 · Abstract (English)
The key-value (KV) cache in transformer models is a critical component for efficient decoding or inference, yet its memory demands scale poorly with sequence length, posing a major challenge for scalable deployment of large language models. Among several approaches to KV cache compression, quantization of key and value activations has been widely explored. Most KV cache quantization methods still need to manage sparse and noncontiguous outliers separately. To address this, we introduce TaDA, a training-free recipe for KV cache compression with quantization precision that adapts to error sensitivity across layers and a mean centering to eliminate separate outlier handling. Our approach yields substantial accuracy improvements for multiple models supporting various context lengths. Moreover, our approach does not need to separately manage outlier elements -- a persistent hurdle in most traditional quantization methods. Experiments on standard benchmarks demonstrate that our technique reduces KV cache memory footprint to 27% of the original 16-bit baseline while achieving comparable accuracy. Our method paves the way for scalable and high-performance reasoning in language models by potentially enabling inference for longer context length models, reasoning models, and longer chain of thoughts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。