提出新权重衰减规则，实现模型宽度间学习率零样本迁移。
Robust Layerwise Scaling Rules by Proper Weight Decay Tuning
- 针对AdamW优化器设计权重衰减与宽度的匹配规则
- 使各层增益在不同宽度下保持不变，避免超参调优
- 适用于LLaMA类模型，支持跨宽度快速迁移
经验缩放定律规定了参数、数据和算力的分配方式，而最大更新参数化（μP）通过均衡早期更新幅度实现宽度间的学习率迁移。然而，在现代尺度不变架构中，训练很快进入优化器主导的稳态，归一化层引入反向尺度敏感性，导致有效学习率依赖于模型宽度，破坏μP迁移效果。本文提出一种针对AdamW的权重衰减缩放规则，以保持各子层增益在不同宽度下的不变性。实验发现，每个矩阵参数的奇异值谱在范数上按√(η/λ)缩放，且形状近似不变；在宽度缩放d下，最高奇异值约按√(η/λ)·d^0.75增长。结合μP的学习率规则η₂ ∝ d⁻¹，推导出权重衰减规则λ₂ ∝ √d，可近似保持子层增益宽度不变。配合向量类参数以η₁ = Θ_d(1)、λ₁ = 0训练，实现学习率与权重衰减从代理宽度到目标宽度的零样本迁移，无需逐宽调优。在LLaMA风格Transformer及最小合成设置中验证该规则，并提供通过匹配最高奇异值诊断子层增益不变性的方法。本工作通过显式控制优化器设定的稳态尺度，将μP扩展至近初始化以外的范围，为AdamW下的宽度鲁棒超参迁移提供实用方案。
原文摘要 · Abstract (English)
Empirical scaling laws prescribe how to allocate parameters, data, and compute, while maximal-update parameterization ($μ$P) enables learning-rate transfer across widths by equalizing early-time update magnitudes. However, in modern scale-invariant architectures, training quickly enters an optimizer-governed steady state where normalization layers create backward scale sensitivity and the effective learning rate becomes width dependent, degrading $μ$P transfer. We address this by introducing a weight-decay scaling rule for AdamW that preserves sublayer gain across widths. Empirically, the singular-value spectrum of each matrix parameter scales in norm as $\sqrt{η/λ}$ with an approximately invariant shape; under width scaling $d$, we observe that the top singular value scales approximately as $\sqrt{η/λ}\cdot d^{0.75}$. Combining this observation with the $μ$P learning-rate rule $η_2\propto d^{-1}$ for matrix-like parameters implies an empirical weight-decay scaling rule $λ_2\propto \sqrt{d}$ that approximately keeps sublayer gains width invariant. Together with vector-like parameters trained at $η_1=Θ_d(1)$ and $λ_1=0$, this yields \emph{zero-shot} transfer of both learning rate and weight decay from proxy to target widths, removing per-width sweeps. We validate the rule on LLaMA-style Transformers and in a minimal synthetic setting, and we provide a simple diagnostic, matching top singular values, to check sublayer-gain invariance. Our results extend $μ$P beyond the near-init regime by explicitly controlling steady-state scales set by the optimizer, offering a practical recipe for width-robust hyperparameter transfer under AdamW.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。