arXiv:2606.09012cs.LGcs.AI2026-06

解释量化训练为何在极端低比特下有效,揭示梯度偏移机制。

Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin

论文配图:Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
图 1 · 摘自论文原文
  • 用几何视角建模权重空间,发现量化会偏离低损失区域。
  • 传统量化易选高损失点,而QAT通过梯度偏置实现恢复。
  • 适用于追求极致压缩的视觉与语言模型研究者。

后训练量化(PTQ)在不重新训练的情况下将全精度模型转为低比特权重,虽效率高但在极端低比特时易失效;量化感知训练(QAT)成本更高但能恢复精度。本文提出统一几何框架,解释了这两种现象:全精度训练沿低损失‘河床’进行,其周围为近似平坦的‘盆地’,离开盆地则损失急剧上升。当量化网格与盆地宽度相当时,局部量化目标(如舍入或基于海森的二阶重构)可能选择高损失的部署点,即使附近存在低损失量化点。此时,基于直通估计器的QAT具有有益偏差——在更新潜伏全精度权重时,评估部署量化权重处的梯度,使梯度感知到山谷边界并产生向内分量,推动后续量化迭代重回盆地。通过局部景观模型形式化该机制,构建了几何式PTQ失败模式,并在局部量化兼容性假设下证明有限时间内的QAT恢复。跨视觉与语言模型、多种神经网络量化方案的实验验证了预测的盆地区域穿越失败及对应的QAT恢复机制。

原文摘要 · Abstract (English)

Post-training quantization (PTQ) converts a trained full-precision model into low-bit weights without task-level retraining, while quantization-aware training (QAT) incorporates quantization into the training loop. Although PTQ is efficient and often accurate at moderate bitwidths, it can fail sharply at aggressive bitwidths; QAT is more expensive but can often recover the lost accuracy. We propose a unified geometric framework that explains both PTQ failure and QAT recovery. We model full-precision training as following a low-loss \emph{river} inside a wider \emph{valley}: a normal neighborhood of the river forms a nearly flat \emph{basin}, while leaving this basin incurs a sharp loss increase. When the quantization grid is comparable to the basin width, local PTQ objectives, including rounding and Hessian-based second-order reconstruction, can select a high-loss deployed quantized point outside the basin even when nearby low-loss quantized points exist. In this regime, straight-through-estimator-based QAT has a useful bias: it evaluates gradients at the deployed quantized weights while updating latent full-precision weights, causing the gradient to sense the valley wall and acquire an inward component that steers subsequent quantized iterates back into the basin. We formalize this mechanism through a local landscape model, construct a geometric PTQ failure mode, and prove finite-time QAT recovery under local quantizer-compatibility assumptions. Experiments across vision and language models under multiple neural-network quantization schemes corroborate the predicted basin-crossing failure of PTQ and the corresponding recovery mechanism of QAT.

量化QAT优化机制模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。