一种更稳定的优化器，通过归一化梯度再加动量提升训练效果。
Adaptive Optimization via Momentum on Variance-Normalized Gradients
- 先对梯度做方差归一化，再应用动量，避免传统方法的耦合问题。
- 在CIFAR-100和语言建模上优于Adam、AdaBelief等，收敛更平滑。
- 适合追求稳定训练和更好泛化的深度学习研究者使用。
我们提出MVN-Grad（基于方差归一化梯度的动量优化器），一种类Adam优化器，通过结合基于方差的归一化与归一化后的动量，提升稳定性与性能。它将每个坐标缩放为梯度不确定性的指数移动平均，并对归一化后的梯度施加动量，消除了标准Adam型更新中陈旧动量与随机归一化之间的跨时间耦合。理论上证明该解耦使单步条件更新方差更小，且对孤立梯度尖峰具有统一有界响应。在低方差场景下，方差归一化可避免二阶矩缩放的符号坍塌，实现加速收敛。此外，在有界梯度随机假设下，我们给出了MVN-Grad的一般非凸收敛保证。在CIFAR-100和类GPT语言建模任务中，其表现匹配或超越Adam、AdaBelief和LaProp，训练更平稳，泛化性能更优，仅需额外一个状态张量。
原文摘要 · Abstract (English)
We introduce MVN-Grad (Momentum on Variance-Normalized Gradients), an Adam-style optimizer that improves stability and performance by combining two complementary ideas: variance-based normalization and momentum applied after normalization. MVN-Grad scales each coordinate by an exponential moving average of gradient uncertainty and applies momentum to the resulting normalized gradients, removing the cross-time coupling between stale momentum and a stochastic normalizer present in standard Adam-type updates. We prove that this decoupling yields smaller one-step conditional update variance than momentum-then-normalize variance methods, and that MVN-Grad has a uniformly bounded response to isolated gradient spikes. In low-variance regimes, we further show that variance normalization avoids sign-type collapse of second-moment scaling and can yield accelerated convergence. Beyond these comparisons, we prove a general nonconvex convergence guarantee for MVN-Grad under bounded-gradient stochastic assumptions. On CIFAR-100 and GPT-style language modeling, MVN-Grad matches or improves on Adam, AdaBelief, and LaProp, delivering smoother training and better generalization at the cost of one additional state tensor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。