通过噪声微调让敏感权重更抗量化误差，提升低精度大模型性能。
Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization
- 用随机扰动识别并弱化敏感权重的梯度影响。
- 在LLaMA2-7B上使4比特量化性能媲美GPTQ。
- 无需特殊处理敏感权重，适合硬件部署优化。
量化是资源受限环境下高效部署大语言模型的关键步骤。然而，先前研究发现模型中部分权重（即异常值）对量化噪声极为敏感。现有方法将这些异常值保留为浮点或高精度格式以维持性能，给混合精度模型的高效硬件部署带来挑战。本文提出一种新方法——噪声扰动微调（NPFT），通过在参数高效微调（PEFT）过程中对异常值添加随机扰动，降低其关于损失函数的海森迹，从而削弱敏感性。该方法有效缓解了异常值对量化误差的影响，使得量化后模型性能显著提升，且无需对异常值进行特殊处理。在OPT和LLaMA模型上，该方法对均匀与非均匀量化器均实现稳定性能提升，并提升推理效率。值得注意的是，在LLaMA2-7B-4bit基准上，最简单的RTN量化器经NPFT后性能可达到与GPTQ相当水平。
原文摘要 · Abstract (English)
Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights' impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose Noise Perturbation Fine-tuning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。