通过通道缩放与旋转，让4比特量化更精准
SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
- 用通道级缩放加哈达玛变换处理激活值异常
- 在多个大模型上使量化后性能提升10%-30%
- 无需额外计算开销，适合部署优化
我们提出SmoothRot，一种新型后训练量化方法，用于提升大型语言模型（LLMs）在4比特量化下的效率。该方法针对激活值极端异常这一关键挑战，结合通道级缩放与哈达玛变换，有效将极端异常值转化为对量化友好的激活值，显著提升量化精度。在主流LLM（LLaMA2 7B、LLaMA3.1 8B、Mistral 7B）上的实验表明，SmoothRot在语言生成和零样本推理任务中，使量化模型与FP16模型之间的性能差距平均缩小10%-30%，且不增加推理延迟。代码已开源。
原文摘要 · Abstract (English)
We present SmoothRot, a novel post-training quantization technique to enhance the efficiency of 4-bit quantization in Large Language Models (LLMs). SmoothRot addresses the critical challenge of massive activation outliers, by integrating channel-wise scaling with Hadamard transformations. Our technique effectively transforms extreme outliers into quantization-friendly activations, significantly improving quantization accuracy. Experiments conducted on popular LLMs (LLaMA2 7B, LLaMA3.1 8B, and Mistral 7B) demonstrate that SmoothRot consistently reduces the performance gap between quantized and FP16 models by approximately 10-30\% across language generation and zero-shot reasoning tasks, without introducing additional inference latency. Code is available at https://github.com/czakop/smoothrot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。