arXiv:2509.22935cs.LGcs.AI2025-09被引 3

提出计算最优的量化训练策略,提升模型精度并节省算力。

Compute-Optimal Quantization-Aware Training

  • 通过实验发现量化训练占比随总算力增加而提高。
  • 提出基于参数字节-令牌数的缩放定律,可预测最优训练分配。
  • 新方法融合学习率衰减与量化训练,显著减少冗余计算。

量化感知训练(QAT)是提升量化神经网络精度的主流方法。以往研究显示,将训练分为全精度(FP)阶段与QAT阶段,比仅使用QAT效果更优。然而,两者之间的算力最优分配仍不明确。本文在8600万至22亿参数的多种模型规模、不同量化位宽及算力预算下进行大量实验,探究不同QAT时长对最终性能的影响。结果表明,与以往认知相反,随着总计算量增加,损失最优的QAT与FP训练比例也上升。我们发现,通过参数字节-令牌数统计,可准确预测广泛模型规模和量化宽度下的最优比例。基于实验数据,推导出损失缩放律,能预测不同算力分配策略及量化位宽下的最优比例与最终性能。利用该规律进行预测并经实验验证,包括在给定内存约束下最优量化位宽的选择,以及不同位宽下QAT精度与全精度模型的对比。此外,提出一种新的冷却与QAT融合方法,将学习率衰减与量化训练联合进行,消除冗余的全精度更新,实现显著算力节约。这些发现为高效QAT规划提供实践指导,使相同算力下训练出更高精度的量化模型。

原文摘要 · Abstract (English)

Quantization-aware training (QAT) is a leading technique for improving the accuracy of quantized neural networks. Previous work has shown that decomposing training into a full-precision (FP) phase followed by a QAT phase yields superior accuracy compared to QAT alone. However, the optimal allocation of compute between the FP and QAT phases remains unclear. We conduct extensive experiments with various compute budgets, QAT bit widths, and model sizes from 86.0M to 2.2B to investigate how different QAT durations impact final performance. We demonstrate that, contrary to previous findings, the loss-optimal ratio of QAT to FP training increases with the total amount of compute. Moreover, the optimal fraction can be accurately predicted for a wide range of model sizes and quantization widths using the tokens-per-parameter-byte statistic. From experimental data, we derive a loss scaling law that predicts both optimal QAT ratios and final model performance across different QAT/FP compute allocation strategies and QAT bit widths. We use the scaling law to make further predictions, which we verify experimentally, including which QAT bit width is optimal under a given memory constraint and how QAT accuracy with different bit widths compares to full-precision model accuracy. Additionally, we propose a novel cooldown and QAT fusion approach that performs learning rate decay jointly with quantization-aware training, eliminating redundant full-precision model updates and achieving significant compute savings. These findings provide practical insights into efficient QAT planning and enable the training of higher-quality quantized models with the same compute budget.

量化训练算力优化模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。