用双二值分解压缩大模型，加法替代乘法，效果优于现有方法。
Addition is almost all you need: Compressing large language models with double binary factorization
- 将权重矩阵分解为两个二值矩阵与缩放向量的乘积，保留加法计算优势。
- 1比特下性能优于主流二值化方法，2比特下媲美最优量化方案。
- 支持逐层非均匀压缩，适配资源受限场景下的模型部署。
二值量化通过将权重矩阵替换为±1的二值矩阵，并以廉价的加法替代昂贵的乘法，为应对大语言模型日益增长的计算与存储需求提供了高效解决方案。然而，严格的±1约束常导致显著精度下降。本文提出双二值分解（DBF），将密集权重矩阵分解为两个二值（符号）矩阵及其对应的缩放向量的乘积。该方法在保持二值表示计算效率的同时，实现了与或优于当前最优方法的压缩率。具体而言，在每权重1比特条件下，DBF优于现有二值化方法；在每权重2比特条件下，其表现可媲美最优量化方法如QuIP#和QTIP。不同于多数压缩技术仅提供有限压缩比选择，DBF可通过调节分解中间维度实现细粒度压缩比控制。基于此优势，我们进一步提出一种基于已有通道剪枝准则的非均匀逐层压缩比估计算法。代码已开源：https://github.com/usamec/double_binary。
原文摘要 · Abstract (English)
Binary quantization approaches, which replace weight matrices with binary matrices and substitute costly multiplications with cheaper additions, offer a computationally efficient approach to address the increasing computational and storage requirements of Large Language Models (LLMs). However, the severe quantization constraint ($\pm1$) can lead to significant accuracy degradation. In this paper, we propose Double Binary Factorization (DBF), a novel method that factorizes dense weight matrices into products of two binary (sign) matrices, each accompanied by scaling vectors. DBF preserves the efficiency advantages of binary representations while achieving compression rates that are competitive with or superior to state-of-the-art methods. Specifically, in a 1-bit per weight range, DBF is better than existing binarization approaches. In a 2-bit per weight range, DBF is competitive with the best quantization methods like QuIP# and QTIP. Unlike most existing compression techniques, which offer limited compression level choices, DBF allows fine-grained control over compression ratios by adjusting the factorization's intermediate dimension. Based on this advantage, we further introduce an algorithm for estimating non-uniform layer-wise compression ratios for DBF, based on previously developed channel pruning criteria. The code is available at: https://github.com/usamec/double_binary
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。