提出加权准确率评估方法,让分类模型更贴合实际成本最小化需求。
Cost-Sensitive Evaluation for Binary Classifiers
- 定义加权准确率WA,直观反映不同误判代价下的模型表现
- 证明在单位成本下最大化WA等价于最小化总分类成本
- 新重加权框架适用于多种场景,比重采样更契合真实部署成本
选择合适的分类器评估指标对模型比较、参数优化和部署决策至关重要,但目前尚无广泛接受的评估范式能直接对齐总分类成本(TCC)最小化。同时,类别不平衡常被视为需独立纠正的问题,可能与TCC最小化目标产生偏差。为此,我们提出:(i) 定义加权准确率(WA),一种具有明确解释意义的二分类器评估指标,是准确率的加权版本;(ii) 提出一个通用重加权框架,用于在成本敏感场景中处理类别不平衡,作为重采样技术的替代方案。该框架适用于可表示为样本相关量线性组合的任意评估指标或损失函数,能实现不同数据集上评估结果的有意义比较,并考虑开发数据集与目标部署数据集之间的差异。在此框架下,我们推导出标准再平衡技术在何种条件下仍与TCC最小化一致,以及何时可能误导。我们证明,在样本无关的单位分类成本下,最大化WA等价于最小化TCC。最后,通过分析多种类别不平衡和成本配置下的真实场景,研究了WA与TCC的相关性,结果显示,在几乎所有测试场景中,WA均保持与TCC的稳健一致性。
原文摘要 · Abstract (English)
Selecting an appropriate evaluation metric for classifiers is crucial for model comparison, parameter optimization, and deployment decisions, yet there is no consensus on a broadly accepted evaluation paradigm explicitly aligned with Total Classification Cost (TCC) minimization. At the same time, class imbalance is often treated as a problem to be corrected \emph{per se}, potentially causing misalignments with TCC minimization. To address these limitations, (\emph{i}) we define Weighted Accuracy (WA), an evaluation metric for binary classifiers with a straightforward interpretation as a weighted version of accuracy and (\emph{ii}) we propose a general reweighting framework for handling class imbalance in cost-sensitive scenarios, providing an alternative to resampling techniques. This framework applies to any evaluation metric or loss function that can be expressed as a linear combination of example-dependent quantities; it enables meaningful comparison of evaluation results obtained on different datasets and accounts for discrepancies between the \emph{development} dataset, used for training, validation, and testing, and the \emph{target} dataset, where the model will be deployed. Within this framework, we derive the conditions under which standard rebalancing techniques remain coherent with TCC minimization, and when they may instead become misleading. We prove that, under example-independent Unit Classification Costs, maximizing WA is equivalent to minimizing TCC. Finally, we analyze the robustness of WA in realistic example-dependent cost scenarios by studying its correlation with TCC across a broad range of class imbalance and cost regimes. The results show that WA maintains robust alignment with TCC across almost all examined scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。