arXiv:2411.01973cs.LG2024-11

提出新指标$C_ρ$,量化分类器预测的可信度与不确定性贡献。

The Certainty Ratio $C_ρ$: a novel metric for assessing the reliability of classifier predictions

  • 基于概率混淆矩阵分解预测的确定性与不确定性成分
  • $C_ρ$在21个数据集上揭示传统指标忽略的关键可靠性信息
  • 适合关注模型可信度的科研与工业应用者

评估分类器性能在机器学习中至关重要,尤其在高风险场景中,预测可靠性直接影响决策。传统指标如准确率和F-score常忽略预测中的不确定性,导致评估失真。本文提出新指标Certainty Ratio ($C_ρ$),通过融合概率混淆矩阵($CM^\star$)并分解预测为确定性与不确定性成分,量化二者对性能指标的贡献。在21个数据集及多种分类器(决策树、朴素贝叶斯、3-最近邻、随机森林)上的实验表明,$C_ρ$揭示了传统指标常忽视的关键可靠性洞察。结果强调将概率信息纳入分类器评估的重要性,为研究者和实践者提供增强模型可信度的有力工具。

原文摘要 · Abstract (English)

Evaluating the performance of classifiers is critical in machine learning, particularly in high-stakes applications where the reliability of predictions can significantly impact decision-making. Traditional performance measures, such as accuracy and F-score, often fail to account for the uncertainty inherent in classifier predictions, leading to potentially misleading assessments. This paper introduces the Certainty Ratio ($C_ρ$), a novel metric designed to quantify the contribution of confident (certain) versus uncertain predictions to any classification performance measure. By integrating the Probabilistic Confusion Matrix ($CM^\star$) and decomposing predictions into certainty and uncertainty components, $C_ρ$ provides a more comprehensive evaluation of classifier reliability. Experimental results across 21 datasets and multiple classifiers, including Decision Trees, Naive-Bayes, 3-Nearest Neighbors, and Random Forests, demonstrate that $C_ρ$ reveals critical insights that conventional metrics often overlook. These findings emphasize the importance of incorporating probabilistic information into classifier evaluation, offering a robust tool for researchers and practitioners seeking to improve model trustworthiness in complex environments.

分类器评估置信度可靠性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。