用噪声标签预训练，让模型学会更复杂的特征。
Impact of Label Noise on Learning Complex Features
- 用噪声标签预训练，引导SGD跳出简单解空间。
- 在多个架构和数据集上，模型学到更多元的复杂特征。
- 不牺牲性能，适合提升模型泛化能力的研究者。
使用随机梯度下降（SGD）训练的神经网络表现出对简单决策边界的归纳偏置，通常收敛到少数函数，难以捕捉复杂特征。这引发了人们对深度模型能否充分学习和表示真实世界数据集的担忧。传统方法如显式正则化、数据增强和架构修改等，大多无法有效促使模型学习多样化特征。本文研究了在多种架构和数据集上，使用噪声标签预训练对SGD动态的影响。实验表明，噪声标签预训练能促进模型学习复杂函数和多样化特征。即使存在噪声，预训练仍能引导梯度下降找到不依赖于简单特征的替代极小值，从而学习更复杂且更广泛的特征集，同时保持性能不受影响。
原文摘要 · Abstract (English)
Neural networks trained with stochastic gradient descent exhibit an inductive bias towards simpler decision boundaries, typically converging to a narrow family of functions, and often fail to capture more complex features. This phenomenon raises concerns about the capacity of deep models to adequately learn and represent real-world datasets. Traditional approaches such as explicit regularization, data augmentation, architectural modifications, etc., have largely proven ineffective in encouraging the models to learn diverse features. In this work, we investigate the impact of pre-training models with noisy labels on the dynamics of SGD across various architectures and datasets. We show that pretraining promotes learning complex functions and diverse features in the presence of noise. Our experiments demonstrate that pre-training with noisy labels encourages gradient descent to find alternate minima that do not solely depend upon simple features, rather learns more complex and broader set of features, without hurting performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。