对比Adam与高斯-牛顿法在基对齐和噪声下的优化性能
Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
- 从基对齐与梯度噪声角度比较Adam与高斯-牛顿对角预条件方法
- 全批量下Adam可优于GN⁻¹与GN⁻½，随机梯度下表现近似GN⁻½
- 理论分析结合凸与非凸任务的实证，适合优化算法研究者
对角预条件器是计算上可行的二阶优化近似方法，在加速深度学习训练方面展现出显著潜力。主流方法包括基于Adam和高斯-牛顿（GN）的方法：前者利用当前梯度统计，是神经网络的默认优化器；后者使用高斯-牛顿矩阵的对角元素，支撑了如Sophia等近期对角优化器。本文从两个关键因素——预条件器中基的选择、小批量梯度噪声的影响——出发，对比这两种方法。通过在二次目标和逻辑回归上分析四个象限情形，发现无论基如何选择，总存在全批量情况下Adam优于GN⁻¹与GN⁻½的实例。相反，在随机设置下，当数据服从高斯假设时，Adam在线性回归中的行为与GN⁻½相似。这些理论结果在凸与非凸目标上的实证研究中得到支持。
原文摘要 · Abstract (English)
Diagonal preconditioners are computationally feasible approximate to second-order optimizers, which have shown significant promise in accelerating training of deep learning models. Two predominant approaches are based on Adam and Gauss-Newton (GN) methods: the former leverages statistics of current gradients and is the de-factor optimizers for neural networks, and the latter uses the diagonal elements of the Gauss-Newton matrix and underpins some of the recent diagonal optimizers such as Sophia. In this work, we compare these two diagonal preconditioning methods through the lens of two key factors: the choice of basis in the preconditioner, and the impact of gradient noise from mini-batching. To gain insights, we analyze these optimizers on quadratic objectives and logistic regression under all four quadrants. We show that regardless of the basis, there exist instances where Adam outperforms both GN$^{-1}$ and GN$^{-1/2}$ in full-batch settings. Conversely, in the stochastic regime, Adam behaves similarly to GN$^{-1/2}$ for linear regression under a Gaussian data assumption. These theoretical results are supported by empirical studies on both convex and non-convex objectives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。