BatchNorm让网络特征更聚类，但不影响稀疏性。
Impact of Batch Normalization on Convolutional Network Representations
- 对比有无BatchNorm的模型，分析隐藏层激活向量的稀疏性和聚类性。
- 有BatchNorm的模型特征聚类更优，但稀疏性差异不显著。
- 适合关注归一化机制如何影响特征表示的研究者。
批量归一化（BatchNorm）是训练深度神经网络中常用的层归一化技术，已被证明能提升模型训练速度和准确率。然而，其作用机制仍是研究热点，存在多种不同观点。本文研究了BatchNorm对网络隐藏表示的影响，即样本在各隐藏层产生的激活值向量。重点考察这些表示的稀疏性以及隐式聚类——即表示之间形成一定程度相似的分组。通过对比使用和不使用BatchNorm训练的图像分类模型，发现一致差异：BatchNorm对表示稀疏性的影响并非影响泛化的关键因素，而使用BatchNorm的模型其表示具有更优越的聚类特性。
原文摘要 · Abstract (English)
Batch normalization (BatchNorm) is a popular layer normalization technique used when training deep neural networks. It has been shown to enhance the training speed and accuracy of deep learning models. However, the mechanics by which BatchNorm achieves these benefits is an active area of research, and different perspectives have been proposed. In this paper, we investigate the effect of BatchNorm on the resulting hidden representations, that is, the vectors of activation values formed as samples are processed at each hidden layer. Specifically, we consider the sparsity of these representations, as well as their implicit clustering -- the creation of groups of representations that are similar to some extent. We contrast image classification models trained with and without batch normalization and highlight consistent differences observed. These findings highlight that BatchNorm's effect on representational sparsity is not a significant factor affecting generalization, while the representations of models trained with BatchNorm tend to show more advantageous clustering characteristics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。