对比5种校准方法，发现新方法在真实数据上显著提升预测概率准确性。
Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods
- 用5种无模型依赖的后处理校准法改进分类概率输出
- Venn-Abers和Beta校准在多数任务中降低对数损失最多
- 现代强模型用传统校准可能反而变差，需谨慎选择方法
我们研究了适用于真实独立同分布表格式数据的模型无关后处理校准方法，重点考察提供分布自由有效性保证的合取与Venn类方法。在TabArena-v0.1基准上，对21种常用分类器（包括线性模型、SVM、树集成模型如CatBoost、XGBoost、LightGBM，以及现代表格式神经网络和基础模型）进行随机分层五折交叉验证，并保留一个测试集。使用五个校准器（等距回归、Platt缩放、Beta校准、Venn-Abers预测器、Pearsonify）在独立校准集上训练后应用于测试预测。通过合适评分规则（对数损失、Brier得分）和诊断指标（Spiegelhalter's Z、ECE、ECI），以及判别能力（AUC-ROC）和标准分类指标评估校准效果。结果表明，跨任务与架构，Venn-Abers预测器在对数损失上平均下降最大，紧随其后的是Beta校准；而Platt缩放效果较弱且不一致。Beta校准在更多任务中改善对数损失，但Venn-Abers极端退化情况更少、极端提升更多。重要的是，常见校准方法（尤其是Platt缩放和等距回归）会对现代强表格式模型的合适评分性能产生系统性损害。整体分类性能通常保持稳定，但校准效果在数据集与架构间差异显著，无方法全面占优。预期上，除Pearsonify外所有方法略微提升准确率，最大增益约0.008%。
原文摘要 · Abstract (English)
We study model-agnostic post-hoc calibration methods intended to improve probabilistic predictions in supervised binary classification on real i.i.d. tabular data, with particular emphasis on conformal and Venn-based approaches that provide distribution-free validity guarantees under exchangeability. We benchmark 21 widely used classifiers, including linear models, SVMs, tree ensembles (CatBoost, XGBoost, LightGBM), and modern tabular neural and foundation models, on binary tasks from the TabArena-v0.1 suite using randomized, stratified five-fold cross-validation with a held-out test fold. Five calibrators; Isotonic regression, Platt scaling, Beta calibration, Venn-Abers predictors, and Pearsonify are trained on a separate calibration split and applied to test predictions. Calibration is evaluated using proper scoring rules (log-loss and Brier score) and diagnostic measures (Spiegelhalter's Z, ECE, and ECI), alongside discrimination (AUC-ROC) and standard classification metrics. Across tasks and architectures, Venn-Abers predictors achieve the largest average reductions in log-loss, followed closely by Beta calibration, while Platt scaling exhibits weaker and less consistent effects. Beta calibration improves log-loss most frequently across tasks, whereas Venn-Abers displays fewer instances of extreme degradation and slightly more instances of extreme improvement. Importantly, we find that commonly used calibration procedures, most notably Platt scaling and isotonic regression, can systematically degrade proper scoring performance for strong modern tabular models. Overall classification performance is often preserved, but calibration effects vary substantially across datasets and architectures, and no method dominates uniformly. In expectation, all methods except Pearsonify slightly increase accuracy, but the effect is marginal, with the largest expected gain about 0.008%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。