训练不稳定性反而让模型找到更平坦的解，提升泛化能力
Training Instabilities Induce Flatness Bias in Gradient Descent
- 通过分析梯度下降中的特征向量旋转现象，揭示不稳定性诱导平坦性偏差
- 学习率越高，特征向量旋转越强，越能探索平坦区域，提升泛化性能
- 该机制适用于随机梯度和Adam优化器，为不稳定训练提供理论支持
经典梯度下降（GD）分析以损失函数海森矩阵的最大特征值定义稳定阈值，低于该阈值时训练稳定且损失单调下降。然而现代深度网络常在该阈值之上取得最佳性能。我们证明此类不稳定性会引发梯度下降的隐式偏置，促使参数趋向损失曲面更平坦的区域，从而改善泛化。其关键机制是特征向量的旋转极性（RPE），即在训练不稳定性下，海森矩阵的主导特征向量发生旋转，且旋转程度随学习率增加而增强，促进探索并可证明引导至更平坦的极小值。该理论框架可扩展至随机梯度下降，其中不稳定性驱动的平坦化效应仍存在且强于小批量噪声。最后，我们在Adam中恢复不稳定性后进一步提升了泛化性能。这些结果共同确立并理解了训练不稳定性在深度学习中的建设性作用。
原文摘要 · Abstract (English)
Classical analyses of gradient descent (GD) define a stability threshold based on the largest eigenvalue of the loss Hessian, often termed sharpness. When the learning rate lies below this threshold, training is stable and the loss decreases monotonically. Yet, modern deep networks often achieve their best performance beyond this regime. We demonstrate that such instabilities induce an implicit bias in GD, driving parameters toward flatter regions of the loss landscape and thereby improving generalization. The key mechanism is the Rotational Polarity of Eigenvectors (RPE), a geometric phenomenon in which the leading eigenvectors of the Hessian rotate during training instabilities. These rotations, which increase with learning rates, promote exploration and provably lead to flatter minima. This theoretical framework extends to stochastic GD, where instability-driven flattening persists and its empirical effects outweigh minibatch noise. Finally, we show that restoring instabilities in Adam further improves generalization. Together, these results establish and understand the constructive role of training instabilities in deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。