深度残差网络的深层效果源于隐式集成,深度增加等价于集成规模扩大。
On residual network depth
- 提出残差展开定理,证明深度增长等价于隐式集成规模扩大。
- 揭示计算路径的组合爆炸导致输出信号增强,解释归一化层必要性。
- 首次从结构出发解释无归一化方法(如SkipInit)的有效性,提供理论依据。
深度残差架构(如ResNet和Transformer)实现了前所未有的模型深度,但为何深度如此有效仍缺乏理论解释。主流观点认为这些网络表现如同多个浅层模型的集成。本文提出一个显式解析公式,验证了这一集成视角:增加网络深度在数学上等价于扩大该隐式集成的规模。进一步分析发现,计算路径的组合增长导致输出信号爆炸式增强,解释了训练深模型中归一化层的历史必要性。这一洞察为归一化层的存在提供了第一性原理解释,并深化了对一系列成功无归一化技术(如SkipInit和Fixup)的理解。以往方法通过优化器分析或与批归一化的类比推导缩放因子,而本文首次从网络内在函数结构出发,证明每个残差模块的缩放可有效抑制组合爆炸,同时作为容量控制机制隐式正则化模型复杂度。
原文摘要 · Abstract (English)
Deep residual architectures, such as ResNet and the Transformer, have enabled models of unprecedented depth, yet a formal understanding of why depth is so effective remains an open question. A popular intuition, following Veit et al. (2016), is that these residual networks behave like ensembles of many shallower models. Our key finding is an explicit analytical formula that verifies this ensemble perspective, proving that increasing network depth is mathematically equivalent to expanding the size of this implicit ensemble. Furthermore, our expansion reveals a hierarchical ensemble structure in which the combinatorial growth of computation paths leads to an explosion in the output signal, explaining the historical necessity of normalization layers in training deep models. This insight offers a first principles explanation for the historical dependence on normalization layers and sheds new light on a family of successful normalization-free techniques like SkipInit and Fixup. However, while these previous approaches infer scaling factors through optimizer analysis or a heuristic analogy to Batch Normalization, our work offers the first explanation derived directly from the network's inherent functional structure. Specifically, our Residual Expansion Theorem reveals that scaling each residual module provides a principled solution to taming the combinatorial explosion inherent to these architectures. We further show that this scaling acts as a capacity controls that also implicitly regularizes the model's complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。