arXiv:2608.07915cs.LG2026-08

通过谱变换编码,让大模型缓存压缩突破2比特瓶颈。

SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding

论文配图:SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding
图 1 · 摘自论文原文
  • 将缓存转到基于自身统计的谱坐标系,识别出关键信息通道。
  • 在长文本任务中实现4倍压缩近乎无损,8倍压缩仍有效,最高达12倍。
  • 无需训练、可直接替换现有系统,适合需要长上下文的部署场景。

大型语言模型在智能体时代需处理长输入,其推理内存主要由键值(KV)缓存占据,该缓存存储每个已读和生成标记的注意力键与值。随着上下文变长,缓存占用显存增加。现有方法通过量化将存储值压缩至近2比特,但质量急剧下降——因少量异常值占据有限级别,其余信息沦为噪声。我们发现:当缓存被旋转至基于自身统计的坐标系后,强相关性消失,少数通道承载绝大部分信息。据此,我们提出SPECTRA,一种无需训练的即插即用编码器,将缓存重编码至该坐标系,并将比特预算集中于关键通道。在Llama-3.1-8B与Qwen2.5-7B上,SPECTRA在长上下文基准下实现4倍压缩近乎无损,8倍压缩表现仍优于均匀量化,最高可达12倍压缩,显著突破2比特瓶颈,使相同显卡支持更长上下文与更大批量。

原文摘要 · Abstract (English)

Large language models (LLMs) increasingly read long inputs in the agentic era, from whole documents and codebases to conversations across many turns. Their inference memory is then dominated by the key-value (KV) cache, the stored attention keys and values of every token the model has read and generated. Because the cache grows with context length and is re-read in full at every generated token, a longer context means more GPU memory. To reduce this cost, most existing methods compress the KV cache by lowering every stored value to the same low precision, a technique known as quantization. They can push this to nearly two bits per value, but rarely further, because quality drops sharply at this 2-bit cliff: four levels are too few for the cache's outlier-heavy values, where a few large entries consume the levels and collapse the rest into noise. A natural remedy is to spend more bits on the channels (feature dimensions) that matter and fewer on the rest, but the raw cache offers no handle: its channels are strongly correlated, so none stands out as more important. Our analysis shows that this handle appears once the cache is rotated into a coordinate system computed from its own statistics, removing these correlations. There, a small fraction of channels carries almost all the information, and spending the budget on those few is far more accurate than spreading it evenly. Guided by this analysis, we develop SPECTRA, a training-free, drop-in codec that re-encodes the cache into this coordinate system and concentrates the bit budget on the channels that carry the signal. On Llama-3.1-8B and Qwen2.5-7B over long-context benchmarks, SPECTRA is near-lossless at 4x compression, competitive at 8x where uniform quantization has collapsed, and reaches up to 12x, pushing usable compression past the 2-bit cliff so the same GPU holds longer contexts and larger batches.

KV缓存压缩量化长上下文

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。