arXiv:2505.18909stat.MLcs.LG2025-05ICML被引 6

揭示标签噪声如何影响特征学习,解释模型为何会过拟合并提出应对策略。

On the Role of Label Noise in the Feature Learning Process

论文配图:On the Role of Label Noise in the Feature Learning Process
图 1 · 摘自论文原文
  • 通过信号-噪声数据分布建模,分析两层卷积网络的训练动态。
  • 发现模型先学信号(泛化好),后过拟合噪声(泛化差)的双阶段过程。
  • 为早停和样本筛选提供理论依据,适合关注鲁棒训练的研究者。

带有噪声标签的深度学习面临重大挑战。本文从特征学习角度,理论上刻画标签噪声的作用。我们考虑一种信号-噪声数据分布,每个样本包含依赖标签的信号和独立于标签的噪声,并严格分析在该数据设置下,两层卷积神经网络的训练动态,以及标签噪声的影响。分析揭示了两个关键阶段:第一阶段,模型完全拟合所有无噪声的干净样本,忽略带噪声标签的样本,此时从干净样本中学习到的信号具有良好的泛化能力;第二阶段,当训练损失趋近收敛时,噪声方向的梯度超过信号方向,导致模型开始过拟合噪声样本,最终记忆噪声,损害泛化性能。此外,我们的分析为两种广泛使用的抗噪声技术——早停与样本选择——提供了理论基础。在合成和真实数据集上的实验验证了理论结论。

原文摘要 · Abstract (English)

Deep learning with noisy labels presents significant challenges. In this work, we theoretically characterize the role of label noise from a feature learning perspective. Specifically, we consider a signal-noise data distribution, where each sample comprises a label-dependent signal and label-independent noise, and rigorously analyze the training dynamics of a two-layer convolutional neural network under this data setup, along with the presence of label noise. Our analysis identifies two key stages. In Stage I, the model perfectly fits all the clean samples (i.e., samples without label noise) while ignoring the noisy ones (i.e., samples with noisy labels). During this stage, the model learns the signal from the clean samples, which generalizes well on unseen data. In Stage II, as the training loss converges, the gradient in the direction of noise surpasses that of the signal, leading to overfitting on noisy samples. Eventually, the model memorizes the noise present in the noisy samples and degrades its generalization ability. Furthermore, our analysis provides a theoretical basis for two widely used techniques for tackling label noise: early stopping and sample selection. Experiments on both synthetic and real-world setups validate our theory.

标签噪声特征学习理论分析过拟合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。