通过筛选梯度显著性，提升模型泛化能力。
Sharpness-Aware Minimization with Z-Score Gradient Filtering
- 按层使用Z-score过滤梯度，保留最显著的成分
- 在CIFAR-10/100、Tiny-ImageNet上测试准确率提升
- 适合对优化稳定性与泛化性能有要求的研究者
深度神经网络在多个领域表现优异，但在优化受微小或噪声梯度影响时仍存在泛化挑战。尖锐感知最小化通过向高曲率方向扰动参数来提升泛化性能，但其使用完整梯度向量，导致微小或噪声分量可能干扰上升步长，使优化器错过最优解。本文提出基于Z-score梯度过滤的尖锐感知最小化方法，对每层梯度进行Z-score过滤，构建掩码仅保留绝对Z-score最大的前百分位成分。百分位阈值$Q_p$决定保留比例，使上升步长聚焦于相对于层均值最显著的方向。该选择性扰动在优化过程中更精准地搜索平坦极小值，同时降低不显著梯度的影响。在ResNet、VGG和视觉变换器等架构下，于CIFAR-10、CIFAR-100和Tiny-ImageNet上的实验表明，该方法持续优于尖锐感知最小化及其变体，提升了测试准确率。代码已开源：https://github.com/YUNBLAK/Sharpness-Aware-Minimization-with-Z-Score-Gradient-Filtering。
原文摘要 · Abstract (English)
Deep neural networks achieve high performance across many domains but can still face challenges in generalization when optimization is influenced by small or noisy gradient components. Sharpness-Aware Minimization improves generalization by perturbing parameters toward directions of high curvature, but it uses the entire gradient vector, which means that small or noisy components may affect the ascent step and cause the optimizer to miss optimal solutions. We propose Z-Score Filtered Sharpness-Aware Minimization, which applies Z-score based filtering to gradients in each layer. Instead of using all gradient components, a mask is constructed to retain only the top percentile with the largest absolute Z-scores. The percentile threshold $Q_p$ determines how many components are kept, so that the ascent step focuses on directions that stand out most compared to the average of the layer. This selective perturbation refines the search toward flatter minima while reducing the influence of less significant gradients. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet with architectures including ResNet, VGG, and Vision Transformers show that the proposed method consistently improves test accuracy compared to Sharpness-Aware Minimization and its variants. The code repository is available at: https://github.com/YUNBLAK/Sharpness-Aware-Minimization-with-Z-Score-Gradient-Filtering
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。