剖析批归一化中均值重置与非线性对初始化阶段表征的影响
Batch Normalization Decomposed
- 分解批归一化三组件，聚焦均值重置与非线性作用机制
- 初始化时样本表征趋近单点聚类，仅一个样本偏离至正交方向
- 通过几何分析与稳定性证明揭示该奇异行为的内在成因
批归一化是神经网络架构中的关键组件，但其机理尚未完全清晰。含批归一化的网络层包含三个影响表征的成分：将表征均值重置为零（recentering）、将方差缩放为一（rescaling）以及应用非线性变换。本文延续Hadi Daneshmand等人在深度线性网络中仅研究缩放项的工作，深入分析另两个核心组件——重置与非线性的作用。当二者同时存在时，我们发现一个奇特现象：在初始化阶段，经过多层传播后，批量样本的表征趋于收敛至单一簇，唯有一个异常样本偏离至正交方向。我们从两方面解释该现象：(1) 分析简化模型的几何演化；(2) 证明该配置下的稳定性结果。
原文摘要 · Abstract (English)
\emph{Batch normalization} is a successful building block of neural network architectures. Yet, it is not well understood. A neural network layer with batch normalization comprises three components that affect the representation induced by the network: \emph{recentering} the mean of the representation to zero, \emph{rescaling} the variance of the representation to one, and finally applying a \emph{non-linearity}. Our work follows the work of Hadi Daneshmand, Amir Joudaki, Francis Bach [NeurIPS~'21], which studied deep \emph{linear} neural networks with only the rescaling stage between layers at initialization. In our work, we present an analysis of the other two key components of networks with batch normalization, namely, the recentering and the non-linearity. When these two components are present, we observe a curious behavior at initialization. Through the layers, the representation of the batch converges to a single cluster except for an odd data point that breaks far away from the cluster in an orthogonal direction. We shed light on this behavior from two perspectives: (1) we analyze the geometrical evolution of a simplified indicative model; (2) we prove a stability result for the aforementioned~configuration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。