arXiv:2410.15361stat.MLcs.LG2024-10ICML被引 14

提出AURC的统计新表达式,提升不确定性评估的准确性与稳定性。

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators

  • 将人口AURC重定义为加权风险函数,理论更清晰。
  • 插件估计器收敛速度达O(√(ln n)/n),偏差小、方差紧。
  • 适用于医疗诊断等高风险场景的模型可靠性评估。

选择性分类器(SC)基于排序的不确定性阈值设定,可用于医疗诊断、自动驾驶和司法系统等安全关键领域。面积下风险-覆盖曲线(AURC)已成为评估SC系统性能的核心指标。本文给出了人口AURC的正式统计表述,提出可解释为加权风险函数的等价形式。通过蒙特卡洛方法,推导出有限样本下的经验AURC插件估计器。相关权重估计器具有一致性,偏差低,均方误差(MSE)紧密有界。插件估计器被证明以速率$\\(mathcal{O}(\\(sqrt{\\ln(n)/n})$收敛,具有统计一致性。在多个数据集、模型架构和置信度评分函数(CSFs)上进行实验,验证了估计器的一致性与有效性,支持对AURC性能的精细调优。

原文摘要 · Abstract (English)

The selective classifier (SC) has been proposed for rank based uncertainty thresholding, which could have applications in safety critical areas such as medical diagnostics, autonomous driving, and the justice system. The Area Under the Risk-Coverage Curve (AURC) has emerged as the foremost evaluation metric for assessing the performance of SC systems. In this work, we present a formal statistical formulation of population AURC, presenting an equivalent expression that can be interpreted as a reweighted risk function. Through Monte Carlo methods, we derive empirical AURC plug-in estimators for finite sample scenarios. The weight estimators associated with these plug-in estimators are shown to be consistent, with low bias and tightly bounded mean squared error (MSE). The plug-in estimators are proven to converge at a rate of $\mathcal{O}(\sqrt{\ln(n)/n})$ demonstrating statistical consistency. We empirically validate the effectiveness of our estimators through experiments across multiple datasets, model architectures, and confidence score functions (CSFs), demonstrating consistency and effectiveness in fine-tuning AURC performance.

不确定性评估统计学习模型可信度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。