从特征空间角度揭示软标签正则化如何提升模型校准与鲁棒性。
Impact of Regularization on Calibration and Robustness: from the Representation Space Perspective
- 从特征空间视角分析正则化对决策边界和梯度方向的影响。
- 发现正则化通过调整特征分布改善了校准与对抗鲁棒性。
- 适合研究模型可靠性、特征空间机制的读者参考。
近期研究表明，使用软标签的正则化技术（如标签平滑、Mixup、CutMix）不仅能提升图像分类准确率，还能缓解因过度自信预测导致的校准偏差，并增强对对抗攻击的鲁棒性。然而，这些改进背后的机制仍不明确。本文从表示空间（即倒数第二层特征空间）的角度提出新解释。通过考察决策边界与特征结构（或表示向量），我们分析了表示空间中的置信度等高线和梯度方向。进一步探讨了正则化对特征分布的调整如何影响这些等高线与方向，揭示出提升校准与鲁棒性的核心机制。研究为高维表示空间中使用软标签训练与正则化的特性提供了新见解。
原文摘要 · Abstract (English)
Recent studies have shown that regularization techniques using soft labels, e.g., label smoothing, Mixup, and CutMix, not only enhance image classification accuracy but also mitigate miscalibration due to overconfident predictions, and improve robustness against adversarial attacks. However, the underlying mechanisms of such improvements remain underexplored. In this paper, we offer a novel explanation from the perspective of the representation space (i.e., the space of the features obtained at the penultimate layer). Based on examination of decision boundaries and structure of features (or representation vectors), our study investigates confidence contours and gradient directions within the representation space. Furthermore, we analyze the adjustments in feature distributions due to regularization in relation to these contours and directions, from which we uncover central mechanisms inducing improved calibration and robustness. Our findings provide new insights into the characteristics of the high-dimensional representation space in relation to training and regularization using soft labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。