首次实现大模型亚1比特量化，可在消费级显卡运行700亿参数模型。
NanoQuant: Efficient Sub-1-Bit Quantization of Large Language Models
- 将权重量化为低秩二值矩阵与缩放因子，结合高效ADMM求解器初始化
- 在单张H100上13小时完成量化，压缩率高达25.8倍，70B模型可跑在8GB显卡
- 适合资源受限场景下大模型部署，代码开源可用
权重量化已成为高效服务大语言模型的标准方法。然而，现有方法难以有效将模型压缩至二值（1比特）或亚1比特水平，要么需要大量数据和计算资源，要么增加额外存储开销。本文提出NanoQuant，首个支持后训练量化（PTQ）的亚1比特压缩方法，可将大模型压缩至二值及更低精度。其将量化建模为低秩二值分解问题，通过高效的交替方向乘子法（ADMM）求解器精确初始化潜在二值矩阵与缩放因子，并利用块级与模型重建过程微调参数。结果在低内存后训练量化中建立了新的帕累托前沿，实现亚1比特压缩。该方法使大规模模型部署成为可能：仅用13小时、单张H100即可将Llama2-70B压缩25.8倍，使其能在8GB消费级显卡上运行。代码已开源于https://github.com/SamsungLabs/NanoQuant。
原文摘要 · Abstract (English)
Weight-only quantization has become a standard approach for efficiently serving large language models (LLMs). However, existing methods fail to efficiently compress models to binary (1-bit) levels, as they either require large amounts of data and compute or incur additional storage. In this work, we propose NanoQuant, the first post-training quantization (PTQ) method to compress LLMs to both binary and sub-1-bit levels. NanoQuant formulates quantization as a low-rank binary factorization problem, and compresses full-precision weights to low-rank binary matrices and scales. Specifically, it utilizes an efficient alternating direction method of multipliers (ADMM) solver to precisely initialize latent binary matrices and scales, and then tunes the initialized parameters through a block and model reconstruction process. Consequently, NanoQuant establishes a new Pareto frontier in low-memory post-training quantization, and enables sub-1-bit compression. NanoQuant makes large-scale deployment feasible on consumer hardware. For example, it compresses Llama2-70B by 25.8$\times$ in just 13 hours on a single H100, enabling a 70B model to operate on a consumer 8 GB GPU. Code is available at https://github.com/SamsungLabs/NanoQuant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。