按实例成本评估分类器，更真实反映误判代价差异。
Instance-Level Costs for Nuanced Classifier Evaluation
- 用每个样本的错误成本加权评估，而非统一计算误差率
- 5%错误率模型可实现1.8%的更低净超额成本
- 适合内容审核、医疗筛查等需区分误判严重性的场景
标准分类对所有错误一视同仁，但在内容审核和医疗筛查中，清晰案例的错误比模糊案例更严重。本文基于上下文老虎机框架提出归一化超额成本（NEC）：按每个样本的代价加权分类错误，当代价均匀时退化为标准误差率。代价可来自标注者投票极差、距离决策阈值的距离或置信度评分。在文本、图像和表格基准上，NEC普遍显著低于误差率——误差率为5%的模型可达到1.8%的NEC，表明多数错误集中在模糊且低成本的样本上。此外，通过损失加权、采样策略或回归引入代价进行训练，效果不一致。本框架提供了一种实用方法来推导和评估实例级误判成本，即使代价敏感训练收益有限。
原文摘要 · Abstract (English)
Standard classification treats all errors equally, but in applications such as content moderation and medical screening, mistakes on clear-cut cases are more costly than errors on ambiguous ones. From a contextual bandit framework, we propose normalized excess cost (NEC), a metric that weighs classification errors by per-example costs and reduces to standard error rate when costs are uniform. Costs can derive from annotator vote margins, distance from decision thresholds, or confidence ratings. Across text, image, and tabular benchmarks, we find that NEC is often substantially lower than error rate: models with 5% error rate can achieve 1.8% NEC, revealing that most mistakes concentrate on ambiguous, low-cost examples. We also find that incorporating costs into training via loss weighting, sampling strategies, or regression yields inconsistent benefits. Our framework provides a practical methodology for deriving and evaluating instance-level misclassification costs, even if cost-sensitive training offers limited benefit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。