用梯度大小判断残差网络是否够深，避免无效加深。
Quantifying Depth Sufficiency in Residual Neural Networks: A First-Order Criterion
- 通过激活梯度在残差空间的投影判断深度是否足够。
- 深度饱和时激活梯度范数趋于低水平，不再有改进空间。
- 适合想优化模型结构、避免盲目堆叠层数的研究者。
如何判断训练好的神经网络是否已具备足够的深度？本文在固定函数保持的残差增长协议下研究此问题，该协议规定了插入位置、残差类型、零输出初始化和零状态一阶更新。定义了一阶残差深度饱和：当所有可接受的插入均不导致严格局部下降时即为饱和。证明了残差非退化性是充要条件：新增深度具有一阶价值仅当条件激活梯度在至少一个可接受的残差切空间上有非零投影。该边界对下降兼容的零状态更新成立，且在保持该切空间的局部重参数化下不变。在残差信号可实现条件下，原始激活梯度消失恰好可认证饱和。在ResNets、GPT-2类模型及持续预训练的Pythia检查点中，最大激活梯度范数随深度增加趋向低信号区域。函数保持的增长方式性能可媲美从头训练。结果支持以激活梯度幅值作为剩余经验一阶深度价值的保守诊断指标。
原文摘要 · Abstract (English)
How can we determine whether a trained neural network is already deep enough? We study this under a fixed function-preserving residual-growth protocol specifying insertion locations, residual families, zero-output initializations, and zero-state first-order updates. We define first-order residual depth saturation as the absence of a strict local decrease from every admissible insertion. We prove residual non-degeneracy is necessary and sufficient: additional depth has first-order value exactly when conditional activation gradients have a nonzero projection onto at least one admissible residual tangent space. This boundary is shared by descent-compatible zero-state updates and invariant under regular local reparameterizations preserving that tangent space. Under residual-signal realizability, raw activation-gradient vanishing exactly certifies saturation. Across ResNets, GPT-2-style models, and continued-pretrained Pythia checkpoints, the maximum activation-gradient norm decreases toward a low-signal regime with depth. Function-preserving growth also achieves converged performance competitive with training from scratch. These results support activation-gradient magnitude as a conservative diagnostic of the remaining empirical first-order value of residual depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。