用索引编码解决大模型量化中的异常值问题，大幅降低比特开销。
ICQuant: Index Coding enables Low-bit LLM Quantization
- 基于异常值统计设计索引编码方案，仅需0.3比特压缩量化范围。
- 2.3比特下使Llama3-70B零样本准确率提升130%~150%。
- 无需微调即可媲美最优微调量化器，适合极致压缩场景。
大型语言模型（LLM）的快速部署凸显了高效低比特后训练量化（PTQ）的需求，因其高内存开销。权重量化中的关键挑战是异常值的存在会扩大量化范围并导致显著误差。尽管已有多种异常值抑制技术，但或无法有效缩小量化范围，或带来相对较高的比特开销。本文提出ICQuant，一种新颖框架，利用异常值统计设计高效的索引编码方案，实现面向异常值的权重仅量化。相比现有技术约需1比特开销才能将量化范围减半，ICQuant仅需约0.3比特；在极端压缩场景（如每权重2-3比特）中优势显著。ICQuant可集成于任意现有量化器之上以消除异常值，提升量化质量。仅使用2.3比特/权重和简单标量量化器，ICQuant使2比特Llama3-70B模型的零样本准确率相对于QTIP和QuIP#提升高达130%和150%，且性能与最佳已知微调量化器（PV-tuning）相当，无需微调。
原文摘要 · Abstract (English)
The rapid deployment of Large Language Models (LLMs) highlights the need for efficient low-bit post-training quantization (PTQ), due to their high memory costs. A key challenge in weight quantization is the presence of outliers, which inflate quantization ranges and lead to large errors. While a number of outlier suppression techniques have been proposed, they either: fail to effectively shrink the quantization range, or incur (relatively) high bit overhead. In this paper, we present ICQuant, a novel framework that leverages outlier statistics to design an efficient index coding scheme for outlier-aware weight-only quantization. Compared to existing outlier suppression techniques requiring $\approx 1$ bit overhead to halve the quantization range, ICQuant requires only $\approx 0.3$ bits; a significant saving in extreme compression regimes (e.g., 2-3 bits per weight). ICQuant can be used on top of any existing quantizers to eliminate outliers, improving the quantization quality. Using just 2.3 bits per weight and simple scalar quantizers, ICQuant improves the zero-shot accuracy of the 2-bit Llama3-70B model by up to 130% and 150% relative to QTIP and QuIP#; and it achieves comparable performance to the best-known fine-tuned quantizer (PV-tuning) without fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。