批归一化不能改善神经网络初始化效果
Batch normalization does not improve initialization
- 通过反例证明批归一化不改善初始权重分布
- 实验显示初始化后性能未因批归一化提升
- 适合关注初始化与训练稳定性的研究者
批归一化是神经网络中最重要的正则化技术之一，能通过中心化网络各层显著改善训练。此前有研究声称批归一化可改善初始化。本文提供反例，证明该说法不成立，即批归一化并不能改善初始化。实验结果表明，在相同初始化条件下，加入批归一化并未带来性能提升，说明其作用主要在训练过程中，而非初始化阶段。
原文摘要 · Abstract (English)
Batch normalization is one of the most important regularization techniques for neural networks, significantly improving training by centering the layers of the neural network. There have been several attempts to provide a theoretical justification for batch ormalization. Santurkar and Tsipras (2018) [How does batch normalization help optimization? Advances in neural information rocessing systems, 31] claim that batch normalization improves initialization. We provide a counterexample showing that this claim s not true, i.e., batch normalization does not improve initialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。