用向量量化压缩大模型键值缓存，大幅节省内存且保持高精度。
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
- 引入向量量化技术，将浮点数缓存转为少量整数索引。
- 在LLaMA3.1-8B上实现82.8%压缩率，性能损失仅1.4%。
- 适合资源受限场景下长文本生成，支持4.3倍更长输出。
大型语言模型（LLMs）上下文长度增长导致键值（KV）缓存变大，限制了在资源受限环境中的部署。此前无训练的缓存压缩方法通常依赖低秩近似或标量量化，难以同时实现高压缩比与高重建保真度。我们提出VQKV，一种全新的无训练方法，引入向量量化（VQ），在保持高模型保真度的前提下获得高度压缩的KV表示，可将数千个浮点数用少数整数索引表示。结果表明，VQKV在LLaMA3.1-8B上实现82.8%的压缩比，同时在LongBench上保留98.6%的基线性能，并在相同内存开销下支持4.3倍的生成长度扩展。
原文摘要 · Abstract (English)
The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches for KV cache compression typically rely on low-rank approximation or scalar quantization, which fail to simultaneously achieve high compression ratios and high reconstruction fidelity. We propose VQKV, a novel, training-free method introducing vector quantization (VQ) to obtain highly compressed KV representations while preserving high model fidelity, allowing for the representation of thousands of floating-point values with just a few integer indices. As a result, VQKV achieves an 82.8\% compression ratio on LLaMA3.1-8B while retaining 98.6\% of the baseline performance on LongBench and enabling 4.3x longer generation length on the same memory footprint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。