批判二分类评估旧方法，推荐用更科学的评分规则。
A Consequentialist Critique of Binary Classification Evaluation: Theory, Practice, and Tools
- 从决策理论出发，主张按阈值和基率评估预测质量。
- 实证发现主流会议仍依赖过时的固定阈值指标。
- 开源工具包降低正确评估门槛，适合医疗等高风险场景使用者。
机器学习支持的决策（如诊断测试或预防性羁押）常需将概率预测转为二分类结果。本文采用决策理论中的后果主义视角，主张评估应关注不同阈值和基率下的预测质量，推荐使用合理的评分规则（如Brier分数和对数损失）。然而，我们对ICML、FAccT、CHIL等主要会议的实证审查显示，仍普遍依赖Top-K或固定阈值评估。为此，我们提出一个决策论框架，明确各类评估指标的适用场景，并发布实用的Python工具包briertools，降低正确评分规则的应用门槛。方法上，我们推导并实现了截断版Brier分数，避免完整积分，更契合有限可解释的阈值范围。理论上，我们将Brier分数与决策曲线分析衔接，回应了(Assel et al., 2017)对合理评分规则临床效用的质疑。
原文摘要 · Abstract (English)
Machine learning-supported decisions, such as ordering diagnostic tests or determining preventive custody, often require converting probabilistic forecasts into binary classifications. We adopt a consequentialist perspective from decision theory to argue that evaluation methods should prioritize forecast quality across thresholds and base rates. This motivates the use of proper scoring rules such as the Brier score and log loss. However, our empirical review of practices at major ML venues (ICML, FAccT, CHIL) reveals a dominant reliance on top-K metrics or fixed-threshold evaluations. To bridge this disconnect, we introduce a decision-theoretic framework that maps evaluation metrics to their appropriate use cases, accompanied by a practical Python package, \texttt{briertools}, which lowers the barrier to applying proper scoring rules in practice. Methodologically, we derive and implement a clipped Brier score variant that avoids full integration and better reflects bounded, interpretable threshold ranges. Theoretically, we reconcile the Brier score with decision curve analysis, directly addressing the critique of (Assel, et al. 2017) regarding the clinical utility of proper scoring rules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。