arXiv:2508.21353cs.LGcs.AI2025-08被引 2

通过动态调整噪声类型,让模型更快找到更稳定的大盆地。

Adaptive Heavy-Tailed Stochastic Gradient Descent

  • 根据训练阶段自动切换重尾与轻尾噪声,模拟边缘稳定性现象。
  • 在MNIST、CIFAR-10等数据集上提升收敛速度与泛化性能。
  • 适合从差初始化开始训练或存在噪声的数据场景。

大规模神经网络优化常因过度依赖训练损失而影响泛化能力。学界普遍认为,宽基底(损失缓慢上升的局部最小区域)能提升对输入或参数微小变化的鲁棒性,从而改善泛化。受梯度噪声天然重尾分布及训练中‘边缘稳定性’现象(曲率先增长后趋于平稳)启发,本文提出自适应重尾随机梯度下降(AHTSGD)。该算法在训练初期注入重尾噪声以增强探索,随曲率稳定逐步转为轻尾噪声。通过动态适配损失景观的尖锐度,AHTSGD加速收敛至宽基底。作为首个基于边缘稳定性调节噪声特性的优化算法,AHTSGD在MNIST、CIFAR-10等基准上持续优于SGD及其他噪声方法,尤其在含噪数据集SVHN上表现显著,可加速劣初值下的早期训练,并在干净与含噪设置下均保持良好泛化能力与学习率鲁棒性。

原文摘要 · Abstract (English)

In the era of large-scale neural network models, optimization algorithms often struggle with generalization due to an overreliance on training loss. One key insight widely accepted in the machine learning community is the idea that wide basins (regions around a local minimum where the loss increases gradually) promote better generalization by offering greater stability to small changes in input data or model parameters. In contrast, sharp minima are typically more sensitive and less stable. Motivated by two key empirical observations - the inherent heavy-tailed distribution of gradient noise in stochastic gradient descent and the Edge of Stability phenomenon during neural network training, in which curvature grows before settling at a plateau, we introduce Adaptive Heavy Tailed Stochastic Gradient Descent (AHTSGD). The algorithm injects heavier-tailed noise into the optimizer during the early stages of training to enhance exploration and gradually transitions to lighter-tailed noise as sharpness stabilizes. By dynamically adapting to the sharpness of the loss landscape throughout training, AHTSGD promotes accelerated convergence to wide basins. AHTSGD is the first algorithm to adjust the nature of injected noise into an optimizer based on the Edge of Stability phenomenon. AHTSGD consistently outperforms SGD and other noise-based methods on benchmarks like MNIST and CIFAR-10, with marked gains on noisy datasets such as SVHN. It ultimately accelerates early training from poor initializations and improves generalization across clean and noisy settings, remaining robust to learning rate choices.

优化算法梯度噪声泛化能力边缘稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。