优化器会影响神经网络的缩放规律，不同优化器导致模型性能随规模增长的速率差异显著。
On the Optimizer Dependence of Neural Scaling Laws
- 在随机特征回归框架中测试五种优化器，发现预条件优化器使缩放指数更大
- 在自然语言数据谱（s≈1.0）下，自然梯度比梯度下降的指数高2.6倍
- 提出谱诊断工具，可预测高级优化器在何种条件下更有效
神经网络缩放律中的指数 $α$（即 $L(N) /propto N^{-α}$）通常被视为由架构和数据决定的常数。我们发现 $α$ 实际上系统性依赖于优化器。在控制变量的随机特征回归实验中，我们测量了五种优化器在六种谱条件下的 $α$。预条件优化器始终带来更陡峭的缩放（更大的 $α$），且 $α$ 的提升在大多数谱范围内递增，于 $s = 1.5$ 达到峰值，在 $s = 2.0$ 仍保持较大值。当 $s \approx 1.0$（典型自然语言数据）时，全自然梯度的 $α\approx 0.31$，而梯度下降仅为 $α\approx 0.12$，前者是后者的2.6倍——在随机特征模型中，这一差距随模型规模翻倍而持续放大。该指数变化是否适用于大规模LLM训练尚不确定，已有证据表明其优势可能随规模衰减。结果表明，缩放律预测应考虑优化器选择，并提供了一种谱诊断方法，以预测先进优化器的收益窗口。
原文摘要 · Abstract (English)
The scaling exponent $α$ in neural scaling laws $L(N) \propto N^{-α}$ is commonly treated as a fixed constant set by architecture and data. We present evidence that $α$ depends systematically on the optimizer. In controlled random-feature regression experiments -- the canonical theoretical framework for neural scaling -- we measure $α$ across five optimizer variants and six spectral conditions. Preconditioned optimizers consistently yield steeper scaling (larger $α$), with the $α$-shift increasing across most of the tested spectral range, peaking near $s = 1.5$, and remaining large at $s = 2.0$. At $s \approx 1.0$ (characteristic of natural language), the full natural gradient achieves $α\approx 0.31$ versus $α\approx 0.12$ for gradient descent -- a $2.6\times$ larger fitted exponent that, within the random-feature model, compounds with each model-size doubling. Whether and how this exponent shift transfers to large-scale LLM training -- where recent evidence suggests the advantage may attenuate with scale -- remains an important open question. Our results imply that scaling-law forecasts should account for optimizer choice, and we provide a spectral diagnostic predicting when advanced optimizers will pay off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。