2bit量化KV缓存，用旋转+修正提升长文本生成质量
KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
- 通过哈达玛旋转降低值向量量化误差，结合线性补偿修正键向量误差
- 在LLaMA、Qwen系列模型上实现2比特量化下生成质量优于或持平基线
- 支持超低精度压缩，且自研内核使推理速度提升最高2.55倍
量化键值（KV）缓存是提升大语言模型（LLM）推理效率的有前景策略。然而，极端低精度量化（如2比特）会在存储的键和值张量中引入显著误差，这些误差经点积注意力机制传播，最终导致生成质量下降。为此，我们提出KVLinC框架，以缓解极低精度下KV缓存量化带来的注意力误差。KVLinC结合哈达玛旋转（减少值向量量化误差）与轻量级线性校正适配器（显式补偿量化键向量引入的误差）。在LLaMA、Qwen2.5和Qwen3模型族上的广泛评估显示，KVLinC始终达到或超越强基线性能，同时实现更高的KV缓存压缩率。此外，我们实现了一个定制注意力内核，相比Flash Attention基线，推理速度最高提升2.55倍，支持高效长上下文LLM推理。
原文摘要 · Abstract (English)
Quantizing the key-value (KV) cache is a promising strategy for improving the inference efficiency of large language models (LLMs). However, aggressive quantization to very low precision (e.g., 2 bits) introduces significant errors in the stored key and value tensors, which propagate through the dot-product attention mechanism and ultimately degrade generation quality. To address this, we propose KVLinC, a framework to mitigate attention errors introduced by KV cache quantization in the extreme low-precision regime. KVLinC combines a Hadamard rotation, which reduces quantization error in values, with lightweight linear correction adapters that explicitly compensate for errors introduced by quantized keys. Across extensive evaluations on the LLaMA, Qwen2.5, and Qwen3 model families, KVLinC consistently matches or surpasses strong baselines while achieving higher KV-cache compression. Furthermore, we implement a custom attention kernel that results in upto 2.55x faster inference compared to Flash Attention baseline, enabling efficient long-context LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。