通过梯度外积矩阵重参数化，提升自适应优化算法收敛速度。
Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization
- 基于期望梯度外积矩阵设计正交变换，改善参数空间表示。
- 在自然数据任务中，新方法使自适应优化器收敛更快。
- 适用于对优化器敏感的机器学习场景，如深度学习训练。
自适应优化算法（如 Adagrad、Adam 及其变体）在机器学习和信号处理中广泛应用。然而，这些方法不具有旋转等变性，简单的参数重定义（即基变换）可能显著影响其收敛性能。目前尚缺乏系统研究如何识别使算法表现最佳的“有利”参数化方式。本文提出一种基于期望梯度外积（EGOP）矩阵的正交重参数化方法，该矩阵可通过全批或随机梯度近似获得。我们证明，在一大类函数上，自适应算法对基选择的敏感性取决于 EGOP 矩阵谱的衰减速率。通过理论分析与实证结果表明，常见机器学习任务中的自然数据通常呈现 EGOP 谱衰减特性，暗示了该重参数化方法的巨大潜力。
原文摘要 · Abstract (English)
Adaptive optimization algorithms -- such as Adagrad, Adam, and their variants -- have found widespread use in machine learning, signal processing and many other settings. Several methods in this family are not rotationally equivariant, meaning that simple reparameterizations (i.e. change of basis) can drastically affect their convergence. However, their sensitivity to the choice of parameterization has not been systematically studied; it is not clear how to identify a "favorable" change of basis in which these methods perform best. In this paper we propose a reparameterization method and demonstrate both theoretically and empirically its potential to improve their convergence behavior. Our method is an orthonormal transformation based on the expected gradient outer product (EGOP) matrix, which can be approximated using either full-batch or stochastic gradient oracles. We show that for a broad class of functions, the sensitivity of adaptive algorithms to choice-of-basis is influenced by the decay of the EGOP matrix spectrum. We illustrate the potential impact of EGOP reparameterization by presenting empirical evidence and theoretical arguments that common machine learning tasks with "natural" data exhibit EGOP spectral decay.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。