arXiv:2409.16071cs.LG2024-09被引 12

用软标签训练分类器,提升模型在小样本和噪声数据下的表现。

Learning with Confidence: Training Better Classifiers from Soft Labels

  • 用类别概率分布代替确定标签,让模型学习标签不确定性
  • 在真实合成数据上,软标签方法优于传统硬标签方法
  • 适合标注不准确或需置信度输出的场景

在监督学习中,模型通常使用硬标签(即明确的类别归属)进行训练,但这种方法忽略了标签本身的内在不确定性。本文研究将标签不确定性以类别概率分布形式(即软标签)纳入训练是否能提升分类模型的预测性能。首先,在模拟实验中验证了软标签学习(SLL)在小样本和不平衡数据下估计模型参数的潜力。随后,采用相同基础分类器,对比多种软标签与硬标签的封装方法。在带有清晰标签的真实合成数据上,所有SLL方法均持续优于硬标签方法。由于现实数据常含噪声,且精确软标签难以获取,本文进一步考察了噪声概率估计对模型性能的影响,并引入四种影响人类标注者误校准的典型噪声类型。结果表明,在多数设定下SLL方法仍优于硬标签方法。最后,在一个包含置信度评分的真实数据集上评估,SLL方法在预测噪声硬标签方面达到传统方法水平,同时提供更准确的置信度估计。

原文摘要 · Abstract (English)

In supervised machine learning, models are typically trained using data with hard labels, i.e., definite assignments of class membership. This traditional approach, however, does not take the inherent uncertainty in these labels into account. We investigate whether incorporating label uncertainty, represented as discrete probability distributions over the class labels -- known as soft labels -- improves the predictive performance of classification models. We first demonstrate the potential value of soft label learning (SLL) for estimating model parameters in a simulation experiment, particularly for limited sample sizes and imbalanced data. Subsequently, we compare the performance of various wrapper methods for learning from both hard and soft labels using identical base classifiers. On real-world-inspired synthetic data with clean labels, the SLL methods consistently outperform hard label methods. Since real-world data is often noisy and precise soft labels are challenging to obtain, we study the effect that noisy probability estimates have on model performance. Alongside conventional noise models, our study examines four types of miscalibration that are known to affect human annotators. The results show that SLL methods outperform the hard label methods in the majority of settings. Finally, we evaluate the methods on a real-world dataset with confidence scores, where the SLL methods are shown to match the traditional methods for predicting the (noisy) hard labels while providing more accurate confidence estimates.

软标签不确定性建模分类器训练置信度估计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。