arXiv:2412.08941cs.LGcs.CV2024-12AAAI被引 14

动态调整梯度裁剪阈值,提升模型对噪声标签的鲁棒性。

Optimized Gradient Clipping for Noisy Label Learning

  • 根据干净与噪声样本梯度比例,动态调节裁剪阈值
  • 在多种噪声类型下均显著降低错误分类率
  • 适合训练含噪声标签的真实数据集场景

已有研究证明,通过约束损失函数关于模型预测概率的梯度,可增强模型对噪声标签的鲁棒性。现有方法通常通过验证集固定最优裁剪阈值,但忽略了训练过程中干净与噪声样本梯度分布的动态变化,限制了模型适应能力。为此,本文提出一种简单有效的优化梯度裁剪(OGC)方法,基于裁剪后干净与噪声梯度的比例,建模两类样本分布,动态调整每步训练的裁剪阈值,有效控制噪声梯度影响。我们还提供了统计分析,验证OGC的抗噪能力。在对称、非对称、实例相关及真实噪声等多类噪声设置下的大量实验表明,该方法显著优于现有方法。

原文摘要 · Abstract (English)

Previous research has shown that constraining the gradient of loss function with respect to model-predicted probabilities can enhance the model robustness against noisy labels. These methods typically specify a fixed optimal threshold for gradient clipping through validation data to obtain the desired robustness against noise. However, this common practice overlooks the dynamic distribution of gradients from both clean and noisy-labeled samples at different stages of training, significantly limiting the model capability to adapt to the variable nature of gradients throughout the training process. To address this issue, we propose a simple yet effective approach called Optimized Gradient Clipping (OGC), which dynamically adjusts the clipping threshold based on the ratio of noise gradients to clean gradients after clipping, estimated by modeling the distributions of clean and noisy samples. This approach allows us to modify the clipping threshold at each training step, effectively controlling the influence of noise gradients. Additionally, we provide statistical analysis to certify the noise-tolerance ability of OGC. Our extensive experiments across various types of label noise, including symmetric, asymmetric, instance-dependent, and real-world noise, demonstrate the effectiveness of our approach.

梯度裁剪噪声标签鲁棒学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。