无需校准的低比特向量量化方法，提升大模型推理效率
NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
- 通过三步变换+哈达玛变换对键值缓存做分布对齐
- 1比特和2比特设置下均优于现有方法，吞吐最高提升3倍
- 适合追求高效推理且无校准数据的场景
大语言模型推理通常内存开销巨大，尤其在大批量和长序列情况下，主要因键值（KV）缓存规模庞大。近期向量量化（VQ）被用于缓解此问题，但现有方法依赖校准数据集，易受分布偏移影响。为此，本文提出NSNQuant，一种无需校准的低比特向量量化技术，专用于KV缓存压缩。该方法通过三步变换——逐标记归一化、逐通道中心化、再次逐标记归一化——结合哈达玛变换，有效将标记分布对齐至标准正态分布。这一对齐使单一可复用码本即可实现鲁棒量化。大量实验表明，NSNQuant在1比特和2比特设置下均持续优于先前方法，具备强泛化能力，相比全精度基线最高可提升3倍吞吐。代码已开源。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache. Vector Quantization (VQ) is recently adopted to alleviate this issue, but we find that the existing approach is susceptible to distribution shift due to its reliance on calibration datasets. To address this limitation, we introduce NSNQuant, a calibration-free Vector Quantization (VQ) technique designed for low-bit compression of the KV cache. By applying a three-step transformation-1) a token-wise normalization (Normalize), 2) a channel-wise centering (Shift), and 3) a second token-wise normalization (Normalize)-with Hadamard transform, NSNQuant effectively aligns the token distribution with the standard normal distribution. This alignment enables robust, calibration-free vector quantization using a single reusable codebook. Extensive experiments show that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings, offering strong generalization and up to 3$\times$ throughput gain over full-precision baselines. Code is available at https://github.com/DHdroid/NSNQuant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。