通过谱去噪与量化实现近无损的KV缓存压缩,节省存储且保持性能。
eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization

- 将KV缓存分解为共享结构与逐令牌残差,用最优奇异值收缩提取共享部分。
- 在2.2比特/条时接近或超过未压缩FP16效果,比现有方法少用近1比特。
- 理论支撑强,适合追求高效推理的模型部署与长序列任务场景。
我们发现变压器注意力头中的键值(KV)缓存可自然分解为一个低秩的共享上下文分量和一个满秩的逐标记残差,该结构符合尖峰随机矩阵模型。这一观察催生了eOptShrinkQ,一种两阶段压缩流程:最优奇异值收缩(eOptShrink)自动提取共享结构,而满足薄壳性质且坐标全局分布的残差则由TurboQuant进行量化——这是一种近期提出的逐向量标量量化器,具有近似最优失真保证。通过恢复标量量化所假设的各向同性,谱去噪消除了对异常值处理和专用内积偏差校正的需求,从而释放出更多位用于提升重建质量。随机矩阵理论提供了三项保障:通过BBP相变实现自动秩选择、残差上内积偏差近乎为零、坐标全局分布确保接近最优量化失真。实验验证了eOptShrinkQ在Llama-3.1-8B与Ministral-8B上的有效性,涵盖三个层面:每头均方误差与内积保真度中,eOptShrinkQ在等效质量下比TurboQuant节省近1比特/条;端到端测试于LongBench(16项任务),其在约2.2比特/条时优于3.0比特的TurboQuant;多针检索任务中,2.2比特的eOptShrinkQ表现接近或超越未压缩的FP16,表明谱去噪在检索密集型任务中可作为有益正则化器。
原文摘要 · Abstract (English)
We show that the key-value (KV) cache in transformer attention heads admits a natural decomposition into a low-rank \emph{shared context} component and a full-rank \emph{per-token} residual, well described by the spiked random matrix model. This observation leads to eOptShrinkQ, a two-stage compression pipeline: optimal singular value shrinkage (eOptShrink) automatically extracts the shared structure, and the residual -- which satisfies the \emph{thin shell property} with delocalized coordinates -- is quantized by TurboQuant~\citep{zandieh2025turboquant}, a recently proposed per-vector scalar quantizer with near-optimal distortion guarantees. By restoring the isotropy that scalar quantization assumes, spectral denoising eliminates the need for both outlier handling and dedicated inner product bias correction, freeing those bits for improved reconstruction. The theoretical grounding in random matrix theory provides three guarantees: automatic rank selection via the BBP phase transition, provably near-zero inner product bias on the residual, and coordinate delocalization ensuring near-optimal quantization distortion. Experimentally, we validate eOptShrinkQ on Llama-3.1-8B and Ministral-8B across three levels: per-head MSE and inner product fidelity, where eOptShrinkQ saves nearly one bit per entry over TurboQuant at equivalent quality; end-to-end on LongBench (16 tasks), where eOptShrinkQ at $\sim$2.2 bits per entry outperforms TurboQuant at 3.0 bits; and multi-needle retrieval, where eOptShrinkQ at 2.2 bits closely matches or exceeds uncompressed FP16, suggesting that spectral denoising can act as a beneficial regularizer for retrieval-intensive tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。