用低秩补偿修复4位量化大模型的精度损失，效果显著。
Low-Rank Correction for Quantized LLMs
- 在未量化激活上加全精度低秩矩阵修正量化误差
- 30%秩可完全弥补4位量化带来的精度下降
- 适合追求极致压缩又想保精度的模型部署场景
针对大语言模型（LLM）后训练阶段的模型压缩问题，本文提出一种新的低秩修正方法，用于校正量化过程中激活值的误差。通过在未量化激活上引入全精度低秩权重矩阵，并联合优化量化权重与额外低秩矩阵，实现权重与激活的共同4比特量化（W4A4）。实验表明，在秩为原权重矩阵10%时，精度损失减少超过50%；当秩达30%时，精度差距基本完全消除。该方法在Llama-2、Llama-3、Phi-3和Mixtral四个主流模型上均取得显著效果。
原文摘要 · Abstract (English)
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of calibration input data. In this work, we introduce a new low-rank approach to correct for quantization errors of \emph{activations} in LLMs: we propose to add low-rank weight matrices in full precision that act on the \emph{unquantized} activations. We then solve a joint optimization problem over the quantized representation of the weights and additional low-rank weight matrices to quantize both weights and activations. We focus on the case of 4-bit weight-and-activation quantization (W4A4). Using ranks equivalent to 10\% of the original weight matrix size, our approach reduces the accuracy gap with the original model by more than 50\%. Using ranks equivalent to 30\% of the original weight matrix, the accuracy gap is closed completely. We demonstrate our results on four recent LLMs, namely Llama-2, Llama-3, Phi-3 and Mixtral models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。