用峰度优化旋转，4比特量化仍保持高精度。
KurTail : Kurtosis-based LLM Quantization
- 基于峰度设计旋转策略，缓解激活值中的异常值影响。
- 4比特量化下MMLU准确率提升13.3%，维基困惑度降低15.5%。
- 仅需单张GPU即可完成，适合消费级硬件部署。
大语言模型量化面临异常值挑战，尤其在4比特量化时，传统均匀量化效果不佳。本文提出KurTail，一种基于峰度的后训练量化（PTQ）方法，通过优化峰度来减少激活值尾部影响。该方法实现权重、激活值及KV缓存的4比特量化，并采用层间优化以保障内存效率。实验显示，相比QuaRot，KurTail在MMLU上提升13.3%，维基困惑度下降15.5%；相较于SpinQuant，MMLU提升2.6%，困惑度降低2.9%，且训练成本显著降低。例如，Llama3-70B的旋转学习在SpinQuant中需至少四张NVIDIA H100 80GB GPU，而本方法仅需单张GPU，更适用于消费级设备。
原文摘要 · Abstract (English)
One of the challenges of quantizing a large language model (LLM) is the presence of outliers. Outliers often make uniform quantization schemes less effective, particularly in extreme cases such as 4-bit quantization. We introduce KurTail, a new post-training quantization (PTQ) scheme that leverages Kurtosis-based rotation to mitigate outliers in the activations of LLMs. Our method optimizes Kurtosis as a measure of tailedness. This approach enables the quantization of weights, activations, and the KV cache in 4 bits. We utilize layer-wise optimization, ensuring memory efficiency. KurTail outperforms existing quantization methods, offering a 13.3\% boost in MMLU accuracy and a 15.5\% drop in Wiki perplexity compared to QuaRot. It also outperforms SpinQuant with a 2.6\% MMLU gain and reduces perplexity by 2.9\%, all while reducing the training cost. For comparison, learning the rotation using SpinQuant for Llama3-70B requires at least four NVIDIA H100 80GB GPUs, whereas our method requires only a single GPU, making it a more accessible solution for consumer GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。