低精度训练导致梯度截断,引发参数指数级膨胀和损失突增。
Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes
- 发现损失突增源于浮点精度限制下的梯度截断。
- 正确类梯度被精确舍零,错误类保留梯度,打破零和约束。
- 该机制解释了后期训练中参数与logit的异常增长现象。
深度神经网络在无正则化长期训练中会出现周期性损失尖峰,称为“弹弓机制”。现有研究多归因于优化内在动力学,但其触发机制尚不明确。本文证明该现象实为浮点数精度限制所致。当训练进入高置信阶段,正确类别logit与其他类别logit之差超过吸收误差阈值时,反向传播中正确类梯度会被精确舍入为零,而错误类梯度仍非零。这破坏了类别间梯度的零和约束,导致分类器层参数更新产生系统性漂移。我们证明该漂移与特征形成正反馈回路,使全局分类器均值与全局特征均值呈指数增长。该机制称为数值特征膨胀(NFI)。NFI可解释弹弓尖峰前的快速范数增长、梯度重现及后续损失突增。进一步表明,即使部分吸收未引发可见尖峰,仍会破坏零和约束并驱动参数范数快速上升。本研究将弹弓机制重释为有限精度训练中的数值动态,并为后期训练中的异常参数增长与logit发散提供了可验证解释。
原文摘要 · Abstract (English)
Deep neural networks exhibit periodic loss spikes during unregularized long-term training, a phenomenon known as the "Slingshot Mechanism." Existing work usually attributes this to intrinsic optimization dynamics, but its triggering mechanism remains unclear. This paper proves that this phenomenon is a result of floating-point arithmetic precision limits. As training enters a high-confidence stage, the difference between the correct-class logit and the other logits may exceed the absorption-error threshold. Then during backpropagation, the gradient of the correct class is rounded exactly to zero, while the gradients of the incorrect classes remain nonzero. This breaks the zero-sum constraint of gradients across classes and introduces a systematic drift in the parameter update of the classifier layer. We prove that this drift forms a positive feedback loop with the feature, causing the global classifier mean and the global feature mean to grow exponentially. We call this mechanism Numerical Feature Inflation (NFI). This mechanism explains the rapid norm growth before a Slingshot spike, the subsequent reappearance of gradients, and the resulting loss spike. We further show that NFI is not equivalent to an observed loss spike: in more practical tasks, partial absorption may not produce visible spikes, but it can still break the zero-sum constraint and drive rapid growth of parameter norms. Our results reinterpret Slingshot as a numerical dynamic of finite-precision training, and provide a testable explanation for abnormal parameter growth and logit divergence in late-stage training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。