重新解读三大优化器，发现它们本质是带特定范数的梯度下降。
Old Optimizer, New Norm: An Anthology
- 去掉指数移动平均后，Adam、Shampoo和Prodigy都等价于特定范数下的最速下降。
- 不同层应分配不同张量范数，如线性层与嵌入层即使维度相同也需不同范数。
- 为神经网络结构精细设计度量空间，有望提升训练稳定性与效率。
深度学习优化器通常基于凸理论或近似二阶理论构建。本文选取Adam、Shampoo和Prodigy三种方法，指出它们均可在不依赖凸性假设的前提下，被理解为纯粹的一阶方法。事实上，在关闭指数移动平均后，每种方法均等价于在特定范数下的最速下降。基于此观察，我们提出新的算法设计空间：根据张量在网络中的角色，为其分配不同的算子范数。例如，尽管线性层与嵌入层具有相同的权重空间$\b{R}^{m imes n}$，但因功能不同，应赋予不同范数。我们希望这种对神经网络架构进行细致度量的思想，能推动更稳定、可扩展且更快的训练算法发展。
原文摘要 · Abstract (English)
Deep learning optimizers are often motivated through a mix of convex and approximate second-order theory. We select three such methods -- Adam, Shampoo and Prodigy -- and argue that each method can instead be understood as a squarely first-order method without convexity assumptions. In fact, after switching off exponential moving averages, each method is equivalent to steepest descent under a particular norm. By generalizing this observation, we chart a new design space for training algorithms. Different operator norms should be assigned to different tensors based on the role that the tensor plays within the network. For example, while linear and embedding layers may have the same weight space of $\mathbb{R}^{m\times n}$, these layers play different roles and should be assigned different norms. We hope that this idea of carefully metrizing the neural architecture might lead to more stable, scalable and indeed faster training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。