用嵌套格点量化提升大模型推理效率，4比特下效果远超现有方法。
NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
- 基于自相似嵌套格点设计低复杂度量化方案，适配各类矩阵乘法。
- Llama-3-8B 4比特量化后在WikiText2上困惑度仅6.6，优于其他方法超过55%。
- 无需修改模型即可部署，适用于大模型推理加速，尤其适合资源受限场景。
后训练量化（PTQ）已成为高效部署大语言模型（LLMs）的关键技术。本文提出NestQuant，一种基于自相似嵌套格点的权重与激活量化新方法。近期研究已从理论上证明此类量化器在低精度矩阵乘法中具有信息论最优性。我们基于戈塞特格点实现了一种低复杂度实用版本，可直接嵌入任意矩阵乘法环节（如自注意力、MLP等）。例如，将Llama-3-8B的权重、键值缓存和激活均量化至4比特，在WikiText2上的困惑度为6.6，相比未量化模型（困惑度6.14）的困惑度差距减少超55%，显著优于当前最优方法Metas SpinQuant（7.3）、OstQuant（7.3）和QuaRot（8.2）。在更大模型（最高70B）及多个主流评估基准上的对比也证实NestQuant具有一致优势。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) has emerged as a critical technique for efficient deployment of large language models (LLMs). This work proposes NestQuant, a novel PTQ scheme for weights and activations that is based on self-similar nested lattices. Recent works have mathematically shown such quantizers to be information-theoretically optimal for low-precision matrix multiplication. We implement a practical low-complexity version of NestQuant based on Gosset lattice, making it a drop-in quantizer for any matrix multiplication step (e.g., in self-attention, MLP etc). For example, NestQuant quantizes weights, KV-cache, and activations of Llama-3-8B to 4 bits, achieving perplexity of 6.6 on wikitext2. This represents more than 55% reduction in perplexity gap with respect to unquantized model (perplexity of 6.14) compared to state-of-the-art Metas SpinQuant (perplexity 7.3), OstQuant (7.3) and QuaRot (8.2). Comparisons on bigger models (up to 70B) and on various LLM evaluation benchmarks confirm uniform superiority of NestQuant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。