用预测概率替代缺失标签，更准确评估少数类内部子概念的性能
Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts
- 用多分类子概念模型预测后验概率代替真实标签作为权重
- 在表格、医疗影像和文本数据上验证，未加权指标易误导结果
- 适合关注少数类内部差异的场景，如医学诊断或公平性分析
类别级评估可能掩盖同一类别内不同子概念间的显著性能差异，导致模型平均表现良好却在特定子群体上失效。已有研究发现，不平衡分类中的常见评价指标对较大的少数子概念存在偏差，而使用真实子概念标签进行效用重加权可缓解此问题；但测试时真实标签通常不可得。本文提出一种实用的效用加权评估方法，将不可获取的子概念标签替换为多分类子概念模型预测的后验概率。评估权重定义为该后验下的期望效用，形成一种软性、考虑不确定性的度量，称为预测加权平衡准确率（pBA）。在表格基准及医疗影像与文本数据集上的实验表明，当类别内异质性存在时，未加权得分可能具有误导性，而pBA在子概念分布不均但非极端的情况下提供更稳定、可解释的评估。代码已公开于：https://anonymous.4open.science/r/correcting-bias-imbalance-9C6C/。
原文摘要 · Abstract (English)
Class-level evaluation can conceal substantial performance disparities across subconcepts within the same class, causing models that perform well on average to fail on specific subpopulations. Prior work has shown that common evaluation measures for imbalanced classification are biased toward larger minority subconcepts and that utility-based reweighting using true subconcept labels can mitigate this bias; however, such labels are rarely available at test time. We introduce a practical utility-weighted evaluation that replaces unavailable subconcept labels with predicted posterior probabilities from a multiclass subconcept model. Evaluation weights are defined as the expected utility under this posterior, yielding a soft, uncertainty-aware metric we call predicted-weighted balanced accuracy (pBA). Experiments on tabular benchmarks as well as medical-imaging and text datasets show that unweighted scores can be misleading under within-class heterogeneity, while pBA provides more stable and interpretable assessments when subconcept distributions are uneven but not pathological. Our code is available at: https://anonymous.4open.science/r/correcting-bias-imbalance-9C6C/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。