用音频压缩技术压缩大模型解码内存，5.5倍压缩率仍保高性能
Residual vector quantization for KV cache compression in large language model
- 将残差向量量化引入KV缓存，分组量化+标准差缩放，无额外可训练参数
- 残差深度8时性能接近原始模型，实现5.5倍压缩（相比半精度）
- 非连续通道分组更优，轻量微调能进一步提升效果，适合部署优化
KV缓存压缩方法主要依赖标量量化来降低解码过程中的内存开销。本文将广泛用于高保真音频压缩的残差向量量化（Residual Vector Quantization）应用于大语言模型（LLM）的KV缓存压缩。仅做最小改动：对任意键或值投影矩阵输出进行标准化缩放，将通道分组后使用同一残差向量量化器逐组量化。采用指数移动平均学习码本，无需输入/输出投影等可训练参数。实验发现，残差深度为8时可恢复大部分未量化模型性能。非连续通道分组优于连续分组，且与轻量微调联合使用可进一步提升效果。整体方法在性能上媲美现有量化技术，结构更简单，相比半精度实现5.5倍压缩。
原文摘要 · Abstract (English)
KV cache compression methods have mainly relied on scalar quantization techniques to reduce the memory requirements during decoding. In this work, we apply residual vector quantization, which has been widely used for high fidelity audio compression, to compress KV cache in large language models (LLM). We adapt the standard recipe with minimal changes to compress the output of any key or value projection matrix in a pretrained LLM: we scale the vector by its standard deviation, divide channels into groups and then quantize each group with the same residual vector quantizer. We learn the codebook using exponential moving average and there are no other learnable parameters including the input and output projections normally used in a vector quantization set up. We find that a residual depth of 8 recovers most of the performance of the unquantized model. We also find that grouping non-contiguous channels together works better than grouping contiguous channels for compressing key matrix and the method further benefits from a light weight finetuning of LLM together with the quantization. Overall, the proposed technique is competitive with existing quantization methods while being much simpler and results in 5.5x compression compared to half precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。