通过正态分布特性实现KV缓存低比特量化，大幅提升大模型推理效率。
NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
- 基于块内元素服从正态分布，采用分块分位数量化降低误差
- 使OPT模型推理可支持2倍大批次或4倍长上下文，吞吐提升9.3倍
- 适合需要高吞吐、长上下文的大模型部署场景
大语言模型在众多任务中表现卓越，但为提升吞吐或满足长上下文需求，常需增大批量大小或延长上下文长度，导致推理时键值（KV）缓存内存消耗显著增加，成为部署瓶颈。针对此问题，量化是常见且直接的解决方案。当前激活量化的上限通常为8比特，进一步降低比特数会引发明显精度下降。为更高效压缩KV缓存，本文分析了其元素分布特性，提出NQKV算法。由于每个块内的元素近似服从正态分布，NQKV采用分块分位数量化，在信息论上达到最优量化误差。在不显著影响输出质量的前提下，NQKV使OPT模型推理支持2倍更大的批量大小或4倍更长的上下文长度，并将吞吐量提升至未使用KV缓存时的9.3倍。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable proficiency across a wide range of tasks. However, LLMs often require larger batch sizes to enhance throughput or longer context lengths to meet task demands, which significantly increases the memory resource consumption of the Key-Value (KV) cache during inference, becoming a major bottleneck in LLM deployment. To address this issue, quantization is a common and straightforward approach. Currently, quantization methods for activations are limited to 8-bit, and quantization to even lower bits can lead to substantial accuracy drops. To further save space by quantizing the KV cache to even lower bits, we analyzed the element distribution of the KV cache and designed the NQKV algorithm. Since the elements within each block of the KV cache follow a normal distribution, NQKV employs per-block quantile quantization to achieve information-theoretically optimal quantization error. Without significantly compromising model output quality, NQKV enables the OPT model to perform inference with an 2x larger batch size or a 4x longer context length, and it improves throughput by 9.3x compared to when the KV cache is not used.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。