提出1比特量化新方法，让大模型压缩后性能超越原版浮点模型。
ARB-LLM: Alternating Refined Binarizations for Large Language Models
- 通过交替优化权重二值化参数，减少量化误差。
- 在LLaMA-7B上实现比现有方法高3.2%的准确率，首次超越同规模浮点模型。
- 适合部署资源受限场景的大模型应用，如移动端或边缘设备。
大语言模型（LLMs）虽推动了自然语言处理的发展，但其高内存与计算需求阻碍实际部署。二值化作为有效的压缩技术，可将模型权重压缩至1比特，显著降低计算与内存开销。然而，现有方法难以缩小二值化权重与全精度权重间的分布差距，且忽略大型语言模型权重的列偏移问题。为此，本文提出面向大模型的新型1比特后训练量化（PTQ）方法ARB-LLM。为缩小分布差异，设计交替精炼二值化（ARB）算法，逐步更新二值化参数，大幅降低量化误差。同时，考虑校准数据与列偏差的影响，进一步扩展出ARB-X与ARB-RC。此外，采用列分组位图（CGB）优化权重划分策略，提升性能。结合CGB后，分别得到ARB-LLM$_\text{X}$与ARB-LLM$_\text{RC}$，显著优于当前最先进的二值化方法。作为1比特后训练量化方法，ARB-LLM$_\text{RC}$首次在相同规模下超越FP16模型。代码与模型将公开于https://github.com/ZHITENGLI/ARB-LLM。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_\text{X}$ and ARB-LLM$_\text{RC}$ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLM$_\text{RC}$ is the first to surpass FP16 models of the same size. The code and models will be available at https://github.com/ZHITENGLI/ARB-LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。