发现归一化训练中学习率与权重衰减的相互作用规律，可精准控制模型训练稳定性。
When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay
- 通过解析模型揭示归一化优化中学习率与权重衰减的精确离散定律
- 在多种网络结构与数据集上验证该定律，性能在边界处达到峰值
- 为自适应优化器的弱稳定现象提供理论解释，适合优化研究者参考
归一化使神经网络大部分区域呈现尺度不变性，导致学习率调度与权重衰减通过参数范数产生隐式反馈回路，共同调控优化器的实际步长。我们证明该交互由一个精确的离散时间定律支配：单一标量量完全捕捉调度与衰减的驱动作用，而范数增长则引发对抗性的几何自抑制效应。这形成清晰的收缩与扩张主导有效学习率区间的分界线。通过解析求解的归一化回归模型（二维动力学），我们发现平衡点本质不稳定，表明恒定学习率配合权重衰减无法稳定维持内部均衡，反而产生由离散时间雅可比结构驱动的周期行为。进一步基于统一齐次优化器框架扩展至多类优化器，揭示自抑制强度的结构性差异，首次从原理上解释为何自适应方法在归一化下系统性表现出更弱的稳定性。在动力系统及多种神经网络（MLP、CNN、GPT2 / MNIST、CIFAR、wikiText、OpenWebText）中，该预测定律以高精度成立，且可直接通过该标量调控训练过程，性能在预测边界处出现尖锐峰值。这些结果确立了尺度不变优化中的单一主导量，为现代深度学习的训练动态、优化器行为与调度设计提供了精确且可操作的视角。代码见 https://github.com/shasanamin/normalized-optimization-dynamics。
原文摘要 · Abstract (English)
Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand the underlying mechanism, we provide exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions and show that the balance point is intrinsically unstable, implying that constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure. We further extend this perspective across optimizers through unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization. Across dynamical systems and neural networks (MLP, CNN, GPT2 / MNIST, CIFAR, wikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary. Together, these results isolate a single governing quantity for scale-invariant optimization, providing a precise and actionable lens on training dynamics, optimizer behavior, and schedule design in modern deep learning. Code is available in https://github.com/shasanamin/normalized-optimization-dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。