提出新优化器AdamO，分离参数大小与方向的更新，提升模型稳定性。
Decoupled Orthogonal Dynamics: Regularization for Deep Network Optimizers
- 将参数幅值和方向分开更新，用SGD式方法控制幅值，自适应预条件仅作用于方向。
- 在图像与语言任务上，相比AdamW显著提升泛化性能与训练稳定性。
- 适用于需要高精度特征学习的场景，尤其适合深层网络和尺度不变层。
AdamW虽将权重衰减与自适应梯度缩放解耦，但仍存在根本矛盾：径向拉锯战。深度学习中，梯度倾向于增大参数范数以扩展有效容量，同时调整方向以学习特征；而权重衰减则无差别抑制范数增长。这种推拉作用引发径向振荡，干扰Adam的二阶矩估计，损害精细的切向特征学习。我们主张幅值与方向应独立处理。提出正交动态解耦，并实现为AdamO：采用类似SGD的更新控制一维范数，而自适应预条件仅限于切向子空间。AdamO还引入曲率自适应径向步长、架构感知规则与投影机制，用于尺度不变层及低维参数。在视觉与语言任务上的实验表明，AdamO在不增加复杂约束的前提下，优于AdamW的泛化性与稳定性。
原文摘要 · Abstract (English)
Is the standard weight decay in AdamW truly optimal? Although AdamW decouples weight decay from adaptive gradient scaling, a fundamental conflict remains: the Radial Tug-of-War. In deep learning, gradients tend to increase parameter norms to expand effective capacity while steering directions to learn features, whereas weight decay indiscriminately suppresses norm growth. This push--pull interaction induces radial oscillations, injecting noise into Adam's second-moment estimates and potentially degrading delicate tangential feature learning. We argue that magnitude and direction play distinct roles and should be decoupled in optimizer dynamics. We propose Orthogonal Dynamics Decoupling and instantiate it as AdamO: an SGD-style update handles the one-dimensional norm control, while Adam's adaptive preconditioning is confined to the tangential subspace. AdamO further incorporates curvature-adaptive radial step sizing and architecture-aware rules and projections for scale-invariant layers and low-dimensional parameters. Experiments on vision and language tasks show that AdamO improves generalization and stability over AdamW without introducing additional complex constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。