用斯坦因收缩改进批归一化，提升对抗攻击下的模型鲁棒性
Admissibility of Stein Shrinkage for Batch Normalization in the Presence of Adversarial Attacks
- 用斯坦因收缩估计批归一化的均值方差，更准确
- 在对抗攻击下，新方法在CIFAR-10等任务上达到当前最优性能
- 降低局部利普希茨常数，增强模型稳定性，适合防御攻击场景
批归一化（BN）是深度神经网络中广泛使用的操作，主要用于提升训练稳定性和正则化效果。它通过样本均值和方差对特征图进行中心化和缩放，而这些统计量天然适用于斯坦因收缩估计。使用该收缩可使均值和方差估计在均方误差意义下更准确。本文证明，在采用次高斯分布建模的对抗攻击下，斯坦因收缩估计量对均值和方差分别优于样本均值与样本方差估计量。此外，基于詹姆斯-斯坦（JS）的批归一化具有更小的局部利普希茨常数，表明其具备更好的正则性质，可能提升鲁棒性。该方法在标准ResNet架构上应用于CIFAR-10图像分类、PPMI（神经影像）数据的3D CNN以及Cityscapes数据上的HRNet图像分割任务中，无论有无对抗攻击，均取得当前最优表现。
原文摘要 · Abstract (English)
Batch normalization (BN) is a ubiquitous operation in deep neural networks, primarily used to improve stability and regularization during training. BN centers and scales feature maps using sample means and variances, which are naturally suited for Stein's shrinkage estimation. Applying such shrinkage yields more accurate mean and variance estimates of the batch in the mean-squared-error sense. In this paper, we prove that the Stein shrinkage estimator of the mean and variance dominates over the sample mean and variance estimators, respectively, in the presence of adversarial attacks modeled using sub-Gaussian distributions. Furthermore, by construction, the James-Stein (JS) BN yields a smaller local Lipschitz constant compared to the vanilla BN, implying better regularity properties and potentially improved robustness. This facilitates and justifies the application of Stein shrinkage to estimate the mean and variance parameters in BN and the use of it in image classification and segmentation tasks with and without adversarial attacks. We present SOTA performance results using this Stein-corrected BN in a standard ResNet architecture applied to the task of image classification using CIFAR-10 data, 3D CNN on PPMI (neuroimaging) data, and image segmentation using HRNet on Cityscape data with and without adversarial attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。