通过随机舍入噪声平滑量化训练损失，让优化器稳定收敛。
LOTION: Smoothing the Optimization Landscape for Quantized Training
- 用随机舍入噪声替代量化损失，构造连续可导的近似表面。
- 在150M和300M语言模型上，性能优于标准量化微调方法。
- 理论保证局部极小值收敛，且全局最优解不丢失。
对量化目标进行神经网络优化本质上具有挑战性，因为量化器是分段常数函数，导致除量化阈值外处处梯度为零，且在阈值处导数未定义。现有大多数方法通过直线传输估计器（STE）等技巧松弛梯度计算，但缺乏收敛性保障。受Nesterov光滑启发，本文提出LOTION（低精度优化通过随机噪声平滑），将量化损失表面近似为连续表面。具体而言，用无偏随机舍入噪声下的期望损失替换原始量化损失。在此框架下，标准优化器可保证收敛至损失面的局部最小值；当使用来自随机舍入的噪声时，原量化损失的全局最小值得以保留。实验表明，该方法在合成测试平台及150M与300M参数语言模型上均优于标准量化微调（QAT）。
原文摘要 · Abstract (English)
Optimizing neural networks for quantized objectives is fundamentally challenging because the quantizer is piece-wise constant, yielding zero gradients everywhere except at quantization thresholds where the derivative is undefined. Most existing methods deal with this issue by relaxing gradient computations with techniques like Straight Through Estimators (STE) and do not provide any guarantees of convergence. In this work, taking inspiration from Nesterov smoothing, we approximate the quantized loss surface with a continuous loss surface. In particular, we introduce LOTION, \textbf{L}ow-precision \textbf{O}ptimization via s\textbf{T}ochastic-no\textbf{I}se sm\textbf{O}othi\textbf{N}g, a principled smoothing framework that replaces the raw quantized loss with its expectation under unbiased randomized-rounding noise. In this framework, standard optimizers are guaranteed to converge to a local minimum of the loss surface. Moreover, when using noise derived from stochastic rounding, we show that the global minima of the original quantized loss are preserved. We empirically demonstrate that this method outperforms standard QAT on synthetic testbeds and on 150M- and 300M- parameter language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。