提出SINQ方法,让低精度大模型无需校准就能减少性能下降。
SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights
- 通过行和列的方差归一化,自动学习每层权重的缩放因子。
- 在4比特下对Qwen3等模型,困惑度降低超过50%。
- 无需额外计算开销,适配新架构,适合部署优化场景。
后训练量化已成为部署大语言模型至低精度的主流方法。然而,当前方法在4比特及以下位宽时仍出现困惑度下降,部分原因是异常值导致与之同尺度的参数精度受损。这一问题在无校准、均匀量化的方案中尤为严重。本文提出SINQ,通过引入第二轴缩放因子,并采用快速的Sinkhorn-Knopp式算法,优化每行每列的方差分布。该方法可近似激活感知量化,从权重矩阵结构中恢复出预测典型激活幅度的列缩放因子。该方法不涉及层间交互,可直接应用于任意线性层和新架构。我们在Qwen3等模型上评估,SINQ在WikiText2和C4数据集上比无校准均匀量化基线降低超过50%的困惑度,计算开销为零或可忽略,且可进一步与校准和非均匀量化结合提升效果。代码已开源于https://github.com/huawei-csl/SINQ。
原文摘要 · Abstract (English)
Post-training quantization has emerged as the most widely used strategy for deploying large language models at low precision. Still, current methods show perplexity degradation at bit-widths less than or equal to 4, partly because representing outliers causes precision issues in parameters that share the same scales as these outliers. This problem is especially pronounced for calibration-free, uniform quantization methods. We introduce SINQ to augment existing post-training quantizers with an additional second-axis scale factor and a fast Sinkhorn-Knopp-style algorithm that finds scales to normalize per-row and per-column variances. We show that this approximates activation-aware quantization by recovering column scales from the weight matrix structure that are predictive of the typical activation magnitudes the matrix received during training. Our method has no interactions between layers and can be trivially applied to new architectures to quantize any linear layer. We evaluate our method on the Qwen3 model family, among others. SINQ reduces the perplexity gap on WikiText2 and C4 by over 50% against uncalibrated uniform quantization baselines, incurs zero to negligible compute overhead, and can be further enhanced by combining it with calibration and non-uniform quantization levels. Code is available at https://github.com/huawei-csl/SINQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。