在低信噪比下,给梯度加标签噪声能抑制过拟合,提升模型泛化能力。
How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?
- 在梯度更新中引入标签噪声,抑制噪声记忆。
- 模型信号快速增长,测试误差显著降低。
- 适合处理含噪声数据的深度学习任务。
深度学习模型容量通常足以同时学习数据中的真实信号和噪声,尤其在低信噪比(SNR)情况下,噪声记忆会严重损害泛化性能。受已有研究启发——标签噪声具有隐式正则化作用并能改善泛化,本文研究在低SNR条件下,通过在梯度下降中加入标签噪声是否可提升神经网络的测试表现。我们考虑在一个理想化的信号-噪声数据设置下训练两层神经网络,采用简单的标签噪声梯度下降(GD)算法。理论证明:训练过程中引入标签噪声可有效抑制噪声记忆,避免其主导学习过程;因此,标签噪声GD能实现快速信号增长,同时控制过拟合,从而在低SNR下仍获得良好泛化。相比之下,标准GD训练的网络在相同低SNR设置下倾向于过拟合噪声,并存在不可消除的测试误差下界,验证了梯度更新中引入标签噪声的有效性。
原文摘要 · Abstract (English)
The capacity of deep learning models is often large enough to both learn the underlying statistical signal and overfit to noise in the training set. This noise memorization can be harmful especially for data with a low signal-to-noise ratio (SNR), leading to poor generalization. Inspired by prior observations that label noise provides implicit regularization that improves generalization, in this work, we investigate whether introducing label noise to the gradient updates can enhance the test performance of neural network (NN) in the low SNR regime. Specifically, we consider training a two-layer NN with a simple label noise gradient descent (GD) algorithm, in an idealized signal-noise data setting. We prove that adding label noise during training suppresses noise memorization, preventing it from dominating the learning process; consequently, label noise GD enjoys rapid signal growth while the overfitting remains controlled, thereby achieving good generalization despite the low SNR. In contrast, we also show that NN trained with standard GD tends to overfit to noise in the same low SNR setting and establish a non-vanishing lower bound on its test error, thus demonstrating the benefit of introducing label noise in gradient-based training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。