提出新型量化方法，实现4比特权重与激活的高精度推理。
LO-BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM Inference
- 将张量分块聚类，为每类设计专用量化码本。
- 在多个LLM上实现W4A4量化后误差低于1%。
- 无需微调，适合部署低比特大模型的场景。
后训练量化（PTQ）是一种无需额外训练成本即可降低大语言模型存储与计算开销的有前景方法。近期研究主要聚焦于仅对权重进行亚8比特量化，而保持激活值在8比特或更高。在不依赖量化感知训练的前提下，实现权重与激活的高精度亚8比特量化仍具挑战性。本文提出一种名为块聚类量化（BCQ）的新方法：将每个操作数张量分解为块（连续标量的组），根据统计特性对块进行聚类，并为每个簇设计专用的最优量化码本。作为该方法的具体实现，我们提出一种迭代优化的后训练量化算法——局部最优块聚类量化（LO-BCQ），通过交替执行块聚类与码本设计，贪婪地最小化量化均方误差。当权重与激活均编码为W4A4格式（存储缩放因子和码本选择器需0.5比特开销）时，我们在多个大语言模型及下游任务上实现了推理精度损失小于1%，显著优于当前最先进水平。
原文摘要 · Abstract (English)
Post-training quantization (PTQ) is a promising approach to reducing the storage and computational requirements of large language models (LLMs) without additional training cost. Recent PTQ studies have primarily focused on quantizing only weights to sub-8-bits while maintaining activations at 8-bits or higher. Accurate sub-8-bit quantization for both weights and activations without relying on quantization-aware training remains a significant challenge. We propose a novel quantization method called block clustered quantization (BCQ) wherein each operand tensor is decomposed into blocks (a block is a group of contiguous scalars), blocks are clustered based on their statistics, and a dedicated optimal quantization codebook is designed for each cluster. As a specific embodiment of this approach, we propose a PTQ algorithm called Locally-Optimal BCQ (LO-BCQ) that iterates between the steps of block clustering and codebook design to greedily minimize the quantization mean squared error. When weight and activation scalars are encoded to W4A4 format (with 0.5-bits of overhead for storing scaling factors and codebook selectors), we advance the current state-of-the-art by demonstrating <1% loss in inference accuracy across several LLMs and downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。