用可控误差ε的软化编码缓解标签噪声,提升模型鲁棒性。
$ε$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise
- 提出ε-softmax,通过可调误差ε近似独热向量以放松对称性约束
- 理论证明可控制过拟合风险,适用于任意损失函数且在真实噪声数据上表现更优
- 结合对称损失实现鲁棒性与拟合能力的平衡,适合存在标签噪声的任务
标签噪声是训练高精度深度神经网络的常见挑战。现有方法多采用稳健损失函数,尤其依赖对称性损失来增强抗噪能力,但常因过于严格的对称条件导致欠拟合问题。本文提出一种简单有效的方案——ε-softmax,仅修改softmax层输出,使其以可控误差ε逼近独热向量,从而松弛对称性要求。ε-softmax不仅可替代传统softmax,还隐式改变损失函数形式。理论上证明其能为几乎所有损失函数提供可控的超额风险界,实现噪声容忍学习。考虑到ε-softmax增强的损失可能削弱在干净数据上的拟合能力,进一步将其与一种对称损失结合,实现鲁棒性与学习效率之间的更好权衡。大量实验表明,该方法在合成及真实世界标签噪声场景下均显著优于现有方法。代码已开源:https://github.com/cswjl/eps-softmax。
原文摘要 · Abstract (English)
Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. In this work, we propose a simple yet effective approach for relaxing the symmetric condition, namely $ε$-softmax, which simply modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $ε$. Essentially, $ε$-softmax not only acts as an alternative for the softmax layer, but also implicitly plays the crucial role in modifying the loss function. We prove theoretically that $ε$-softmax can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that $ε$-softmax-enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method in mitigating synthetic and real-world label noise. The code is available at https://github.com/cswjl/eps-softmax.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。