提升主动学习的预测可信度，让模型更准确地判断自己多不确定。
Calibrated Uncertainty Sampling for Active Learning
- 先选最不靠谱的样本（高校准误差），再用深度网络不确定性判断
- 实验显示在多种场景下校准误差和泛化误差均低于现有方法
- 适合需要可靠置信度评估的医疗、金融等高风险应用
我们研究如何在主动学习中训练出校准误差低的分类器。目前主流的主动学习获取函数（AF）依赖模型对未标记数据的不确定性进行查询，但若该不确定性本身未经校准，将严重影响查询效果，导致泛化性能差且在新数据上校准误差高。深度神经网络（DNN）的不确定性通常未校准，问题更严重。为此，我们提出一种新获取函数：先估计未标记池中样本的校准误差，选择校准误差最高的样本，再利用DNN不确定性进行最终查询。我们采用核校准误差估计器，在协变量偏移下证明了该方法能保证未标记池及未知测试数据上的校准误差有界。实验表明，该方法在各类池基主动学习设置中均显著降低校准误差与泛化误差。
原文摘要 · Abstract (English)
We study the problem of actively learning a classifier with a low calibration error. One of the most popular Acquisition Functions (AFs) in pool-based Active Learning (AL) is querying by the model's uncertainty. However, we recognize that an uncalibrated uncertainty model on the unlabeled pool may significantly affect the AF effectiveness, leading to sub-optimal generalization and high calibration error on unseen data. Deep Neural Networks (DNNs) make it even worse as the model uncertainty from DNN is usually uncalibrated. Therefore, we propose a new AF by estimating calibration errors and query samples with the highest calibration error before leveraging DNN uncertainty. Specifically, we utilize a kernel calibration error estimator under the covariate shift and formally show that AL with this AF eventually leads to a bounded calibration error on the unlabeled pool and unseen test data. Empirically, our proposed method surpasses other AF baselines by having a lower calibration and generalization error across pool-based AL settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。