为神经网络训练设计自适应动量与非线性阻尼，提升稳定性与收敛速度。
Adaptive Momentum and Nonlinear Damping for Neural Network Training
- 为每个参数动态调节动量系数，依据其动能变化自动调整。
- 在ViT、BERT、GPT2上表现优于mSGD，接近或超过Adam性能。
- 理论证明方法具有指数级收敛性，适合复杂模型优化场景。
动量随机梯度下降（mSGD）采用全局固定动量系数，难以适应现代损失曲面的异质结构。本文基于连续时间建模，为每个模型参数引入由其动能决定的自适应动量系数，实现对训练动态的自动调节，在保持收敛速度的同时增强稳定性。我们发现这种自适应摩擦机制与结构动力学中的立方阻尼密切相关。进一步地，通过在mSGD和Adam的连续动力学中加入立方阻尼项，提出两种新优化算法。实验表明，这些方法在ViT、BERT和GPT2任务中展现出强鲁棒性，性能达到或超越Adam，而传统mSGD在此类任务中表现不佳。理论分析证明了所提方法的指数收敛性。
原文摘要 · Abstract (English)
Momentum Stochastic Gradient Descent (mSGD) relies on a fixed momentum coefficient shared across all parameters, failing to account for the heterogeneous structure of modern loss landscapes. In this work, we adopt a continuous-time formulation to introduce individual, adaptive momentum coefficients regulated by the kinetic energy of each model parameter. This mechanism automatically adjusts to evolving training dynamics to maintain stability without sacrificing convergence speed. We demonstrate that this adaptive friction is inextricably linked to cubic damping, a suppression mechanism from structural dynamics. We additionally introduce two optimization schemes by augmenting the continuous dynamics of mSGD and Adam with a cubic damping term. Empirically, our methods demonstrate robustness and match or outperform Adam on training ViT, BERT, and GPT2 tasks where mSGD typically struggles. We further provide theoretical results establishing the exponential convergence of the proposed schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。