低精度训练会压缩梯度，导致SGD收敛变慢。
Why Does Stochastic Gradient Descent Slow Down in Low-Precision Training?
- 将低精度训练视为梯度缩放，分析其对SGD的影响
- 证明收敛速度受最小缩放因子制约，误差更高
- 适合关注模型训练效率与量化影响的研究者
低精度训练对降低大规模深度学习的计算和内存开销至关重要。然而，梯度量化会引入幅度缩小，改变随机梯度下降（SGD）的收敛行为。本文研究在梯度缩放模型下的SGD收敛性，其中每个随机梯度被缩放因子 $ q_k \in (0,1] $ 缩放。我们发现，这种缩放相当于将原步长 $ μ_k $ 变为有效步长 $ μ_k q_k $，当 $ q_{\min} < 1 $ 时会减缓收敛。在典型光滑性和有界方差假设下，证明低精度SGD仍收敛，但速度由 $ q_{\min} $ 决定，且稳态误差因量化效应升高。理论分析揭示了数值精度降低如何通过梯度缩放减慢训练过程。
原文摘要 · Abstract (English)
Low-precision training has become crucial for reducing the computational and memory costs of large-scale deep learning. However, quantizing gradients introduces magnitude shrinkage, which can change how stochastic gradient descent (SGD) converges. In this study, we explore SGD convergence under a gradient shrinkage model, where each stochastic gradient is scaled by a factor \( q_k \in (0,1] \). We show that this shrinkage affect the usual stepsize \( μ_k \) with an effective stepsize \( μ_k q_k \), slowing convergence when \( q_{\min} < 1 \). With typical smoothness and bounded-variance assumptions, we prove that low-precision SGD still converges, but at a slower pace set by \( q_{\min} \), and with a higher steady error level due to quantization effects. We analyze theoretically how lower numerical precision slows training by treating it as gradient shrinkage within the standard SGD convergence setup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。