用梯度定位关键权重，实现高效低比特大模型压缩
GWQ: Gradient-Aware Weight Quantization for Large Language Models
- 根据梯度信息识别重要权重，仅用少量校准数据定位异常值
- 保留顶部1%异常值为16位浮点，其余权重低比特存储，性能更优
- 适合资源受限设备部署，推理速度提升1.2倍，内存显著降低
大语言模型在复杂语言任务中表现优异，但参数量巨大，部署面临挑战。为解决此问题，本文提出梯度感知权重量化（GWQ），首个利用梯度定位异常值的低比特量化方法，仅需少量校准数据即可有效检测异常权重。GWQ优先保留前1%异常值的FP16精度，其余非异常权重采用低比特存储。我们在语言建模、视觉-语言问答、多任务理解等任务上广泛评估，结果表明GWQ优于现有量化方法。量化过程中仅需一个校准集，且相比原模型实现1.2倍推理加速，显著降低推理内存占用。
原文摘要 · Abstract (English)
Large language models (LLMs) show impressive performance in solving complex language tasks. However, its large number of parameters presents significant challenges for the deployment. So, compressing LLMs to low bits can enable to deploy on resource-constrained devices. To address this problem, we propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization that leverages gradients to localize outliers, requiring only a minimal amount of calibration data for outlier detection. GWQ retains the top 1\% outliers preferentially at FP16 precision, while the remaining non-outlier weights are stored in a low-bit. We widely evaluate GWQ on different task include language modeling, grounding detection, massive multitask language understanding and vision-language question and answering. Results show that models quantified by GWQ performs better than other quantization method. During quantization process, GWQ only need one calibration set to realize effective quant. Also, GWQ achieves 1.2x inference speedup in comparison to the original model and effectively reduces the inference memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。