AlphaGrad通过非线性归一化优化器提升训练稳定性，适合内存受限场景。
AlphaGrad: Non-Linear Gradient Normalization Optimizer
- 用L2归一化加tanh变换实现梯度尺度不变性，仅需调节一个参数α。
- 在PPO中表现显著优于Adam，TD3中需调参才能稳定，DQN则不稳定。
- 适合内存受限的强化学习任务，尤其对在线策略学习有明显优势。
我们提出AlphaGrad，一种内存高效、条件无状态的优化器，解决Adam等自适应方法的内存开销与超参数复杂性问题。通过逐张量L2梯度归一化和平滑双曲正切变换 $g' = \tanh(α\cdot \tilde{g})$ 实现尺度不变性，由单个陡度参数 $α$ 控制。贡献包括：（1）提出AlphaGrad算法；（2）提供非凸收敛分析，保证驻点收敛；（3）在多种强化学习基准（DQN、TD3、PPO）上进行广泛实验。相比Adam，AlphaGrad性能高度依赖上下文：在离线策略DQN中不稳定，在TD3中需精细调α以获稳定且竞争性结果，在在线策略PPO中表现显著更优。结果凸显α选择的决定性作用，揭示优化器动态与底层算法间的强耦合。AlphaGrad为内存受限场景提供有力替代方案，在在线策略学习中其稳定性和效率优势尤为突出。
原文摘要 · Abstract (English)
We introduce AlphaGrad, a memory-efficient, conditionally stateless optimizer addressing the memory overhead and hyperparameter complexity of adaptive methods like Adam. AlphaGrad enforces scale invariance via tensor-wise L2 gradient normalization followed by a smooth hyperbolic tangent transformation, $g' = \tanh(α\cdot \tilde{g})$, controlled by a single steepness parameter $α$. Our contributions include: (1) the AlphaGrad algorithm formulation; (2) a formal non-convex convergence analysis guaranteeing stationarity; (3) extensive empirical evaluation on diverse RL benchmarks (DQN, TD3, PPO). Compared to Adam, AlphaGrad demonstrates a highly context-dependent performance profile. While exhibiting instability in off-policy DQN, it provides enhanced training stability with competitive results in TD3 (requiring careful $α$ tuning) and achieves substantially superior performance in on-policy PPO. These results underscore the critical importance of empirical $α$ selection, revealing strong interactions between the optimizer's dynamics and the underlying RL algorithm. AlphaGrad presents a compelling alternative optimizer for memory-constrained scenarios and shows significant promise for on-policy learning regimes where its stability and efficiency advantages can be particularly impactful.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。