系统对比标签噪声检测方法,找出最优组合。
Benchmarking noisy label detection methods
- 拆解方法为三部分:收集策略、不一致度量、聚合方式
- 在多数场景下,样本内+平均概率聚合+对数几率差效果最好
- 提出新评估指标,适合真实噪声场景的检测应用
标签噪声是现实数据集中的常见问题,影响模型训练与评估。尽管已有多种噪声标签检测技术,但尚无明确最优方案。本文将检测方法分解为三个基础组件:收集策略(样本内 vs 样本外)、标签不一致度量和聚合方法,该框架可适配多数现有方法,实现系统性比较。为公平评估,提出统一任务:检测等于数据集噪声率比例的样本。引入新指标:固定操作点下的假阴性率。在视觉与表格数据集上,覆盖合成与真实噪声场景进行评估。结果表明,在多数情况下,采用样本内收集、平均概率聚合,结合对数几率差作为不一致度量,表现最佳。研究结果为设计新方法及特定场景选型提供实用指导。
原文摘要 · Abstract (English)
Label noise is a common problem in real-world datasets, affecting both model training and validation. Clean data are essential for achieving strong performance and ensuring reliable evaluation. While various techniques have been proposed to detect noisy labels (or label errors), there is no clear consensus on optimal approaches. We perform a comprehensive benchmark of detection methods by decomposing them into three fundamental components: gathering strategy (in-sample vs out-of-sample), label disagreement measure, and aggregation method. This decomposition can be applied to many existing detection methods, and enables systematic comparison across diverse approaches. To fairly compare methods, we propose a unified benchmark task: detecting a fraction of training samples equal to the dataset's noise rate. We also introduce a novel metric: the false negative rate at this fixed operating point. Our evaluation spans vision and tabular datasets under both synthetic and real-world noise conditions. We identify that in-sample gathering using average probability aggregation combined with the logit margin as the label disagreement measure achieves the best results across most scenarios. Our findings provide practical guidance for designing new detection methods and selecting techniques for specific applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。