45个数据集验证:单一数据集结论不可靠,阈值调优在不平衡数据中普遍有效
When Single-Dataset Conclusions Fail: A 45-Task Study of Threshold Tuning and Resampling for Imbalanced Classification

- 在45个二分类任务上统一评估阈值调优与重采样方法
- 随机森林在多数任务中通过调优提升F1达0.101,欺诈数据集结果具误导性
- 校准误差无法预测调优收益,实践者需谨慎依赖直观判断
类别不平衡处理通常仅在单个基准数据集上评估,所得结论被当作方法的普适属性。我们表明该做法存在风险。在公开的Kaggle信用卡欺诈数据集上,采用无泄漏的嵌套交叉验证,随机森林在默认0.5阈值下F1为0.861±0.021,阈值调优未带来改善(ΔF1 = -0.002)。仅凭此结论易得‘对校准良好的集成模型,不平衡处理无需’的错觉。但将相同协议应用于45个二分类任务(不平衡比1:1.5至1:178,共2,025次模型拟合,四种模型家族),结论反转:随机森林在全集上显著受益于阈值调优(ΔF1 = +0.101±0.134),其他三类模型也表现出类似行为。SMOTE在欺诈数据集上损害性能,但在全集上平均提升ΔF1 = +0.076(138胜39负,Wilcoxon p = 2.7e-17)。进一步发现:阈值调优收益与不平衡比非单调相关——低于1:5时几乎无益,1:15–1:40区间峰值达+0.120,超过1:100后降至+0.045;这解释了为何欺诈数据集(1:577)代表性不足。此外,验证集校准误差不能预测调优收益(预期校准误差相关系数r = -0.087,Brier r = +0.137),故校准诊断无法指导实践者是否应进行调优。本文发布完整评估协议、45项任务测试集及所有运行指标。
原文摘要 · Abstract (English)
Class-imbalance handling is routinely evaluated on a single benchmark dataset, and the resulting conclusions are reported as if they were properties of the method. We show this practice is unsafe. On the public Kaggle credit-card fraud dataset, under a leakage-free nested cross-validation protocol in which the decision threshold is selected on a held-out inner validation fold, a plain Random Forest at the default 0.5 threshold attains F1 = 0.861 +/- 0.021, and threshold tuning yields it no benefit (delta-F1 = -0.002). Read alone, this supports an appealing conclusion: for a well-calibrated ensemble, imbalance handling is unnecessary. We then apply the identical protocol to 45 binary tasks spanning imbalance ratios from 1:1.5 to 1:178 (2,025 model fits, four model families). The conclusion reverses. Random Forest benefits most from threshold tuning across the suite (delta-F1 = +0.101 +/- 0.134), not least, while three other families replicate their fraud-dataset behaviour almost exactly. SMOTE likewise harms the fraud dataset but helps across the suite (mean delta-F1 = +0.076; 138 wins, 39 losses; Wilcoxon p = 2.7e-17). Two further results. Threshold-tuning benefit is non-monotonic in the imbalance ratio: near zero below 1:5, peaking at +0.120 in the 1:15-1:40 band, declining to +0.045 beyond 1:100 - explaining why the fraud dataset, at 1:577, is an unrepresentative place to study the question. And we reject an intuitive heuristic: validation-set calibration error does not predict tuning benefit (expected calibration error r = -0.087; Brier r = +0.137), so calibration diagnostics cannot tell a practitioner whether tuning is worthwhile. We release the protocol, the 45-task harness, and all per-run metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。