提出统一学习率缩放法则,让卷积网和残差网无需调参即可深度扩展。
Arithmetic-Mean $μ$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets
- 用层间平均更新量约束替代逐层固定,适配异构网络结构
- 发现学习率与深度呈-3/2次方关系,跨深度保持有效
- 可零样本迁移学习率,适用于主流卷积与残差架构
选择合适的学习率仍是扩展现代深度网络深度的关键挑战。经典的最大更新参数化(μP)对同质多层感知机(MLPs)有效,但在残差连接和卷积引入层间不平衡的异构架构中变得不适用。本文提出算术平均μP(AM-μP),不再约束每层更新幅度,而是将全网络单步前激活二阶矩的平均值控制在恒定尺度。结合残差感知的He fan-in初始化——按块数缩放残差分支权重(Var[W] = c/(K·fan-in)),AM-μP实现了宽度鲁棒的深度律,且跨深度一致传递。理论证明:在一维和二维卷积网络中,最大更新学习率满足η⋆(L) ∝ L⁻³/²;带零填充时,边界效应在N≫k下为常量级。对于含通用卷积+MLP模块的标准残差网络,η⋆(L) = Θ(L⁻³/²),其中L为最小深度。大量实验验证了-3/2缩放规律,并实现零样本学习率迁移,为卷积网络与深度残差网络提供统一、实用的学习率原则,无需额外调参开销。
原文摘要 · Abstract (English)
Choosing an appropriate learning rate remains a key challenge in scaling depth of modern deep networks. The classical maximal update parameterization ($μ$P) enforces a fixed per-layer update magnitude, which is well suited to homogeneous multilayer perceptrons (MLPs) but becomes ill-posed in heterogeneous architectures where residual accumulation and convolutions introduce imbalance across layers. We introduce Arithmetic-Mean $μ$P (AM-$μ$P), which constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale. Combined with a residual-aware He fan-in initialization - scaling residual-branch weights by the number of blocks ($\mathrm{Var}[W]=c/(K\cdot \mathrm{fan\text{-}in})$) - AM-$μ$P yields width-robust depth laws that transfer consistently across depths. We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies $η^\star(L)\propto L^{-3/2}$; with zero padding, boundary effects are constant-level as $N\gg k$. For standard residual networks with general conv+MLP blocks, we establish $η^\star(L)=Θ(L^{-3/2})$, with $L$ the minimal depth. Empirical results across a range of depths confirm the $-3/2$ scaling law and enable zero-shot learning-rate transfer, providing a unified and practical LR principle for convolutional and deep residual networks without additional tuning overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。