arXiv:2510.04988cs.LG2025-10被引 1

提出动态调整动量的自适应记忆机制,提升深度学习优化效果。

Adaptive Memory Momentum via a Model-Based Framework for Deep Learning Optimization

  • 用双平面近似目标函数,动态生成随训练变化的动量系数。
  • 在多种任务中超越固定动量的SGD和AdamW,无需额外调参。
  • 方法简洁通用,适合需要高效优化的深度学习场景。

现代深度学习模型大多采用基于动量的一阶优化器。动量项决定了优化器的记忆能力,即历史梯度对当前收敛方向的影响程度。经典方法如Nesterov加速梯度和Heavy Ball,以及近期的AdamW和Lion,均采用固定的动量系数β=0.9,虽被广泛使用,但并非最优。本文提出一种自适应记忆机制,用在线更新的动态动量系数替代固定值。方法基于两个平面近似目标函数:一个来自当前迭代梯度,另一个来自历史梯度累积记忆。据我们所知,此类邻近框架首次用于动量优化。该方法新颖、实现简单,无需额外假设或超参数调优。我们在多种学习任务中实现了SGD与AdamW的自适应版本,涵盖从简单凸问题到大规模深度学习场景,结果表明其性能优于手动调优动量的标准SGD和AdamW。本工作为优化中的自适应机制开辟了新路径。

原文摘要 · Abstract (English)

The vast majority of modern deep learning models are trained with momentum-based first-order optimizers. The momentum term governs the optimizer's memory by determining how much each past gradient contributes to the current convergence direction. Fundamental momentum methods, such as Nesterov Accelerated Gradient and the Heavy Ball method, as well as more recent optimizers such as AdamW and Lion, all rely on the momentum coefficient that is customarily set to $β= 0.9$ and kept constant during model training, a strategy widely used by practitioners, yet suboptimal. In this paper, we introduce an \textit{adaptive memory} mechanism that replaces constant momentum with a dynamic momentum coefficient that is adjusted online during optimization. We derive our method by approximating the objective function using two planes: one derived from the gradient at the current iterate and the other obtained from the accumulated memory of the past gradients. To the best of our knowledge, such a proximal framework was never used for momentum-based optimization. Our proposed approach is novel, extremely simple to use, and does not rely on extra assumptions or hyperparameter tuning. We implement adaptive memory variants of both SGD and AdamW across a wide range of learning tasks, from simple convex problems to large-scale deep learning scenarios, demonstrating that our approach can outperform standard SGD and Adam with hand-tuned momentum coefficients. Finally, our work opens doors for new ways of inducing adaptivity in optimization.

优化算法自适应动量深度学习SGD

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。