让神经网络特征更接近正态分布，提升模型泛化与鲁棒性。
On the Importance of Gaussianizing Representations
- 用幂变换和加性噪声设计新归一化层，强制特征服从正态分布。
- 在多种模型、数据集上均提升泛化性能，对宽度、深度、批量大小鲁棒。
- 可替代现有归一化层，适合需要增强稳定性的场景。
正态分布是信息论中的核心分布——它既是最佳信号也是最差噪声，具有最大表征能力，并使联合分布中的不相关性等价于独立性。尽管调整深层神经网络各层激活的均值和方差已显著促进训练，但很少有明确方案说明这些激活应遵循何种分布及如何实现。受正态分布的信息论性质启发，本文提出正态化归一化（Normality Normalization）：一种新型归一化层，利用幂变换促使特征表示趋于正态分布，并在训练中引入加性高斯噪声。实验全面验证了该方法的有效性：在多种主流模型与数据集组合上表现出优异泛化性能；对模型宽度、深度及训练批次大小等常见变化因素具有强鲁棒性；可替代传统归一化层使用；并能有效提升模型对随机扰动的鲁棒性。
原文摘要 · Abstract (English)
The normal distribution plays a central role in information theory - it is at the same time the best-case signal and worst-case noise distribution, has the greatest representational capacity of any distribution, and offers an equivalence between uncorrelatedness and independence for joint distributions. Accounting for the mean and variance of activations throughout the layers of deep neural networks has had a significant effect on facilitating their effective training, but seldom has a prescription for precisely what distribution these activations should take, and how this might be achieved, been offered. Motivated by the information-theoretic properties of the normal distribution, we address this question and concurrently present normality normalization: a novel normalization layer which encourages normality in the feature representations of neural networks using the power transform and employs additive Gaussian noise during training. Our experiments comprehensively demonstrate the effectiveness of normality normalization, in regards to its generalization performance on an array of widely used model and dataset combinations, its strong performance across various common factors of variation such as model width, depth, and training minibatch size, its suitability for usage wherever existing normalization layers are conventionally used, and as a means to improving model robustness to random perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。