梳理深度学习优化器发展脉络，解析核心更新规则与设计思想。
Development of Deep Learning Optimizers: Approaches, Concepts, and Update Rules
- 按时间线解析SGD到AdamW、Sophia等优化器的更新机制
- 系统总结各优化器的关键变量与默认超参数设置
- 适合想理解优化器原理或寻找研究方向的读者
深度学习优化器是使深度神经网络能够学习的关键算法，其性能直接影响训练效果。随着深度学习的快速发展，涌现出多种不同方法的优化器。本文对近年来受关注的优化器进行了综述，从随机梯度下降（SGD）到最新提出的Momentum、AdamW、Sophia和Muon，按时间顺序逐一分析。详细阐述了每种优化器的更新规则，并解释其相关概念与变量。同时讨论了各类优化器所采用的技术、对优化过程的贡献及其默认超参数配置。此外，文章还指出当前深度学习优化中存在的开放挑战，为理解现有优化器状态及探索未来发展方向提供全面参考。
原文摘要 · Abstract (English)
Deep learning optimizers are optimization algorithms that enable deep neural networks to learn. The effectiveness of learning is highly dependent on the optimizer employed in the training process. Alongside the rapid advancement of deep learning, a wide range of optimizers with different approaches have been developed. This study aims to provide a review of various optimizers that have been proposed and received attention in the literature. From Stochastic gradient descent to the most recent ones such as Momentum, AdamW, Sophia, and Muon in chronological order, optimizers are examined individually, and their distinctive features are highlighted in the study. The update rule of each optimizer is presented in detail, with an explanation of the associated concepts and variables. The techniques applied by these optimizers, their contributions to the optimization process, and their default hyperparameter settings are also discussed. In addition, insights are offered into the open challenges encountered in the optimization of deep learning models. Thus, a comprehensive resource is provided both for understanding the current state of optimizers and for identifying potential areas of future development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。