深度残差网络训练中，深度越大越像无限宽网络，误差可精确控制。
The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram
- 用均值微分方程建模深度残差网络训练动态，揭示其深层行为规律。
- 在最优缩放下，模型输出与极限的误差为 $O(1/L + 1/ ext{sqrt}(LM))$，且可证明紧致。
- 适用于研究深层网络训练机制的理论工作者，尤其关注梯度动力学与泛化性能者。
我们研究从标准随机初始化开始的大深度残差网络（ResNets）的梯度训练。结果表明，当深度 $L$ 趋于无穷时，无论实际宽度如何，无限深度的 ResNets 表现如同无限宽网络。具体地，在固定嵌入维度 $D$ 下，随着深度 $L$ 增大，训练动态收敛到唯一的神经均值微分方程（Mean ODE）。当残差尺度为 $Θ_D(α/(LM))$ 且 $α=Θ_D(1)$ 时，模型输出与极限之间的误差为 $O_D(1/L + 1/\sqrt{LM})$，此时均值微分方程表现出非线性参数化特征，即局部特征更新达到最大。相反，当 $α→∞$ 时进入懒惰（lazy）ODE regime，均值微分方程为线性参数化，并给出了对应收敛速率。进一步分析两层感知机块的 ResNet，发现保证最大局部更新的必要充分残差尺度为 $O(\sqrt{D}/(LM))$，并证明高概率误差界为 $O(1/L + \sqrt{D}/\sqrt{LM})$。所有结论基于新视角：初始化随机性使前向与反向传播成为均值微分方程的随机逼近，而传播混沌（单位渐近独立）确保该行为在训练中保持。实验验证所有速率均为紧致。
原文摘要 · Abstract (English)
We study the gradient-based training of large-depth residual networks (ResNets) from standard random initializations. We show that infinite-depth ResNets behave as if they were infinitely wide, regardless of their actual width. More precisely, we obtain that with a fixed embedding dimension $D$, the training dynamics converges to a unique Neural Mean ODE training dynamics as the depth $L$ diverges, regardless of the scaling of the hidden width $M$. For a residual scale $Θ_D\big(\fracα{LM}\big)$ with $α=Θ_D(1)$, we obtain the error bound $O_D\big(\frac{1}{L}+ \frac{1}{\sqrt{LM}}\big)$ between the model's output and its limit after a fixed number gradient of steps. In this regime, the limit exhibits maximal local feature updates, i.e. the Mean ODE is genuinely non-linearly parameterized. In contrast, we show that $α\to \infty$ yields a lazy ODE regime where the Mean ODE is linearly parameterized, and we derive a convergence rate in this case as well. We then focus on the particular case of ResNets with two-layer perceptron blocks, for which we study how these scalings depend on the embedding dimension $D$. We identify the residual scale $O\big(\frac{\sqrt{D}}{LM}\big)$ as necessary and sufficient for maximal local feature updates. In this regime, we prove a high-probability error bound $O\big(\frac{1}{L}+ \frac{\sqrt{D}}{\sqrt{LM}}\big)$ between the ResNet and its limit after a fixed number of gradient steps. Our convergence results rely on a novel mathematical perspective on ResNets : (i) due to the randomness of the initialization, the forward and backward pass through the ResNet behave as the stochastic approximation of certain mean ODEs, and (ii) by propagation of chaos (that is, asymptotic independence of the units) this behavior is preserved through the training dynamics. We verify empirically that all our rates are tight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。