arXiv:2411.11924cs.LGcs.CV2024-11被引 5

用数据蒸馏自动去除真实数据中的标签噪声,提升模型鲁棒性。

Dataset Distillers Are Good Label Denoisers In the Wild

  • 用数据蒸馏技术直接清除标签噪声,避免传统方法的误差循环。
  • 在对称噪声和自然噪声下效果显著,准确率提升最高达12.3%。
  • 适合高隐私要求场景,且无需在线处理,适合部署在敏感数据环境。

从带噪数据中学习已成为深度模型适应真实应用的关键。传统方法通常先评估噪声,再采取丢弃、加权或重标注等策略,但若初始噪声评估不准,易陷入恶性循环。为此,本文提出利用数据蒸馏进行噪声清除的新方法,避免反馈循环,提升训练效率,并通过离线处理提供强隐私保护。我们系统评估了三种代表性蒸馏方法(DATM、DANCE、RCIG)在对称噪声、非对称噪声及真实自然噪声下的表现。实验发现,数据蒸馏在随机噪声场景中可有效去噪,但在结构化非对称噪声下可能将噪声特征吸收进蒸馏样本;此外,长尾类别中困难样本在蒸馏过程中可能发生有损压缩。尽管存在局限,结果表明该方法在高隐私环境中具有显著潜力,尤其适用于噪声普遍存在的实际部署场景。源码已开源:https://github.com/Kciiiman/DD_LNL。

原文摘要 · Abstract (English)

Learning from noisy data has become essential for adapting deep learning models to real-world applications. Traditional methods often involve first evaluating the noise and then applying strategies such as discarding noisy samples, re-weighting, or re-labeling. However, these methods can fall into a vicious cycle when the initial noise evaluation is inaccurate, leading to suboptimal performance. To address this, we propose a novel approach that leverages dataset distillation for noise removal. This method avoids the feedback loop common in existing techniques and enhances training efficiency, while also providing strong privacy protection through offline processing. We rigorously evaluate three representative dataset distillation methods (DATM, DANCE, and RCIG) under various noise conditions, including symmetric noise, asymmetric noise, and real-world natural noise. Our empirical findings reveal that dataset distillation effectively serves as a denoising tool in random noise scenarios but may struggle with structured asymmetric noise patterns, which can be absorbed into the distilled samples. Additionally, clean but challenging samples, such as those from tail classes in imbalanced datasets, may undergo lossy compression during distillation. Despite these challenges, our results highlight that dataset distillation holds significant promise for robust model training, especially in high-privacy environments where noise is prevalent. The source code is available at https://github.com/Kciiiman/DD_LNL.

数据去噪数据蒸馏标签噪声隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。