研究优化器在量化下的表现,发现传统指标失效,Shampoo最抗量化干扰。
Beyond Outliers: A Study of Optimizers Under Quantization
- 对比6种优化器在量化前后的性能,发现峰值误差指标不靠谱。
- 训练时用Shampoo优化器,量化后准确率下降最少,适合低精度部署。
- 首次揭示量化训练中的优化器效率规律,适合模型压缩研究者。
随着新优化器兴起和模型量化成为高效部署的标配,一个关键问题浮现:优化器的选择如何影响量化下的模型性能?尽管两领域均有进展,但优化器与量化交互的系统性证据仍有限。为此,我们研究了不同优化器在量化下的鲁棒性,涵盖后训练量化(PTQ)和量化感知训练(QAT)。首先,使用6种优化器训练50M至1.5B参数的全精度模型,建立调优基线;随后施加PTQ,评估性能退化。发现基于异常值的指标(如最大均值比MMR、峰度Kurtosis)无法有效预测不同优化器的PTQ表现。理论分析表明,这些指标仅捕捉孤立层误差,忽视误差在网络中累积传播。为研究QAT退化,我们从头训练量化模型并对比原精度基线,发现预训练表现优的优化器在QAT下未必最优,而使用Shampoo训练的模型退化最小。最后,我们推导出不同优化器下量化感知训练的缩放规律,证明Shampoo在参数效率上优于所有测试优化器。
原文摘要 · Abstract (English)
As new optimizers gain traction and model quantization becomes standard for efficient deployment, a key question arises: how does the choice of optimizer affect model performance in the presence of quantization? Despite progress in both areas, systematic evidence on optimizer-quantization interactions remains limited. To fill this gap, we study the impact of optimizer choice on model robustness under quantization, considering both post-training quantization (PTQ), and quantization-aware training (QAT). We first train full-precision models, ranging from 50M to 1.5B parameters, with six optimizers, to explore the hyperparameter landscape, and establish well-tuned baselines. We then apply PTQ to evaluate how model performance degrades when trained with different optimizers. We find that outlier-related metrics, such as the max-to-mean ratio (MMR) and Kurtosis, fail to predict the PTQ performance across different optimizers. We show analytically that this is due to the MMR capturing only isolated layer errors, while ignoring how quantization errors accumulate and propagate through the network. To study the QAT degradation, we train quantized models from scratch and compare them to our original-precision baselines. We find that optimizers performing well in the original pretraining setup may not remain optimal under QAT, and that models trained with Shampoo show the lowest accuracy degradation. Finally, we derive scaling laws for quantization-aware training under different optimizers, showing that Shampoo achieves the highest parameter efficiency of all tested optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。