arXiv:2510.04327cs.LGstat.ML2025-10

提出统一学习率缩放法则,让卷积网和残差网无需调参即可深度扩展。

Arithmetic-Mean $μ$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets

  • 用层间平均更新量约束替代逐层固定,适配异构网络结构
  • 发现学习率与深度呈-3/2次方关系,跨深度保持有效
  • 可零样本迁移学习率,适用于主流卷积与残差架构

选择合适的学习率仍是扩展现代深度网络深度的关键挑战。经典的最大更新参数化(μP)对同质多层感知机(MLPs)有效,但在残差连接和卷积引入层间不平衡的异构架构中变得不适用。本文提出算术平均μP(AM-μP),不再约束每层更新幅度,而是将全网络单步前激活二阶矩的平均值控制在恒定尺度。结合残差感知的He fan-in初始化——按块数缩放残差分支权重(Var[W] = c/(K·fan-in)),AM-μP实现了宽度鲁棒的深度律,且跨深度一致传递。理论证明:在一维和二维卷积网络中,最大更新学习率满足η⋆(L) ∝ L⁻³/²;带零填充时,边界效应在N≫k下为常量级。对于含通用卷积+MLP模块的标准残差网络,η⋆(L) = Θ(L⁻³/²),其中L为最小深度。大量实验验证了-3/2缩放规律,并实现零样本学习率迁移,为卷积网络与深度残差网络提供统一、实用的学习率原则,无需额外调参开销。

原文摘要 · Abstract (English)

Choosing an appropriate learning rate remains a key challenge in scaling depth of modern deep networks. The classical maximal update parameterization ($μ$P) enforces a fixed per-layer update magnitude, which is well suited to homogeneous multilayer perceptrons (MLPs) but becomes ill-posed in heterogeneous architectures where residual accumulation and convolutions introduce imbalance across layers. We introduce Arithmetic-Mean $μ$P (AM-$μ$P), which constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale. Combined with a residual-aware He fan-in initialization - scaling residual-branch weights by the number of blocks ($\mathrm{Var}[W]=c/(K\cdot \mathrm{fan\text{-}in})$) - AM-$μ$P yields width-robust depth laws that transfer consistently across depths. We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies $η^\star(L)\propto L^{-3/2}$; with zero padding, boundary effects are constant-level as $N\gg k$. For standard residual networks with general conv+MLP blocks, we establish $η^\star(L)=Θ(L^{-3/2})$, with $L$ the minimal depth. Empirical results across a range of depths confirm the $-3/2$ scaling law and enable zero-shot learning-rate transfer, providing a unified and practical LR principle for convolutional and deep residual networks without additional tuning overhead.

学习率残差网络深度扩展缩放律

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。