针对大模型量化中的层间差异,提出自适应分层变换方法,提升低比特精度。
Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models
- 按层动态选择最优变换类型,突破传统统一变换限制
- 在LLaMA-3-8B上实现4.58点困惑度降低和2.11%零样本准确率提升
- 基于峰度与异常值分析,高效替代耗时的搜索过程
大语言模型部署需大量计算资源,量化是实际应用的关键。然而,激活值和权重中的系统性异常值导致模型性能显著下降,尤其在低比特设置下。现有基于变换的方法(如仿射、旋转)虽能缓解异常值问题,但采用全局统一变换策略,忽视了模型各层分布的异质性。本文提出一种自适应分层变换选择框架,通过将变换选择建模为可微优化问题,实现每层精准匹配最优变换类型。为降低计算开销,我们发现权重分布峰度与最佳变换类型存在关联,提出基于鲁棒z-score归一化的异常值引导层选择方法,在性能接近可微搜索的前提下大幅减少计算成本。在LLaMA系列模型上的实验表明,该方法持续优于主流固定变换方案。例如,在极端W3A3K2V2量化设置下,对LLaMA-3-8B模型,本方法相比当前最优方法FlatQuant,困惑度降低4.58点,六任务平均零样本准确率提升2.11%,证明异质变换选择对最优量化至关重要。
原文摘要 · Abstract (English)
Large language models require significant computational resources for deployment, making quantization essential for practical applications. However, the main obstacle to effective quantization lies in systematic outliers in activations and weights, which cause substantial LLM performance degradation, especially at low-bit settings. While existing transformation-based methods like affine and rotation transformations successfully mitigate outliers, they apply the homogeneous transformation setting, i.e., using the same transformation types across all layers, ignoring the heterogeneous distribution characteristics within LLMs. In this paper, we propose an adaptive transformation selection framework that systematically determines optimal transformations on a per-layer basis. To this end, we first formulate transformation selection as a differentiable optimization problem to achieve the accurate transformation type for each layer. However, searching for optimal layer-wise transformations for every model is computationally expensive. To this end, we establish the connection between weight distribution kurtosis and accurate transformation type. Specifically, we propose an outlier-guided layer selection method using robust $z$-score normalization that achieves comparable performance to differentiable search with significantly reduced overhead. Comprehensive experiments on LLaMA family models demonstrate that our adaptive approach consistently outperforms the widely-used fixed transformation settings. For example, our method achieves an improvement of up to 4.58 perplexity points and a 2.11% gain in average six-task zero-shot accuracy under aggressive W3A3K2V2 quantization settings for the LLaMA-3-8B model compared to the current best existing method, FlatQuant, demonstrating the necessity of heterogeneous transformation selection for optimal LLM quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。