揭示深度线性网络训练动态，解释宽度与学习率的协同效应。
Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer
- 从随机初始化出发，理论刻画大宽度网络梯度下降过程。
- 发现宽度越大越优，且超参数可跨宽度迁移，与经典神经正切参数化不同。
- 适用于研究模型宽度、深度与数据结构对训练的影响，适合机器学习理论研究者。
我们从理论上刻画了在大规模随机数据上，从随机初始化开始的大宽度深度线性网络的梯度下降动态。理论揭示了均值场/最大更新参数化网络中的“越宽越好”效应以及超参数迁移现象，这与神经正切参数化中最优学习率随模型宽度变化的情形形成对比。我们给出了非残差和残差网络的渐近描述，其中残差结构在分支按 $1/ ext{depth}^{1/2}$ 缩放时可实现无限深度极限。此外，还比较了单遍随机梯度下降与每轮重复使用全部数据的训练动态。最后，该模型恢复了近期工作中在丰富区间观察到的幂律结构数据下的加速幂律训练动态。
原文摘要 · Abstract (English)
We theoretically characterize gradient descent dynamics in deep linear networks trained at large width from random initialization and on large quantities of random data. Our theory captures the ``wider is better" effect of mean-field/maximum-update parameterized networks as well as hyperparameter transfer effects, which can be contrasted with the neural-tangent parameterization where optimal learning rates shift with model width. We provide asymptotic descriptions of both non-residual and residual neural networks, the latter of which enables an infinite depth limit when branches are scaled as $1/\sqrt{\text{depth}}$. We also compare training with one-pass stochastic gradient descent to the dynamics when training data are repeated at each iteration. Lastly, we show that this model recovers the accelerated power law training dynamics for power law structured data in the rich regime observed in recent works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。