用二维矩阵拆解分类器概率质量，区分校准与判别能力。
The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Quality
- 基于Z统计量和AUC预期排名划分四类诊断模型
- 实证发现多数模型属'校准差判别弱'或'判别强校准差'
- 建议先优化判别力再调校准，避免盲目最小化整体误差
Brier得分混淆了概率预测的两个独立属性：可靠性（校准误差）与分辨力（判别能力）。我们提出曼奥金概率矩阵，一种类似BCG的二维诊断框架，将二者分离。通过斯皮格尔哈尔特Z统计量和AUC-ROC预期排名，将分类器置于2×2网格中，并归为四类：鹰（两者皆优）、公牛（判别强但校准差）、懒猫（校准好但判别弱）、鼹鼠（两者皆差）。每类对应不同改进策略。我们在涵盖21个分类器、5种后处理校准器和30个真实二分类任务的大型实证研究中填充该矩阵，使用TabArena-v0.1数据集。分类结果明确：CatBoost、TabICL、EBM、TabPFN、GBC、随机森林为鹰；XGBoost、LightGBM、HGB为公牛；Venn-Abers校准在公牛上使对数损失降低6.5%至12.6%，但在鹰类模型上恶化2.1%；SVM、LR、LDA及经验基线预测为懒猫；MLP、KNN、朴素贝叶斯、ExtraTrees为鼹鼠。理论表明：无序保持的后处理校准无法提升判别力（命题1），故校准可修复，判别力难提升。实践原则是：未分解前勿优化整体Brier得分；应先优化判别力，再事后校准。代码与原始实验数据见https://github.com/valeman/classifier_calibration。
原文摘要 · Abstract (English)
The Brier score conflates two distinct properties of probabilistic predictions: reliability (calibration error) and resolution (discriminatory power). We introduce the Manokhin Probability Matrix, a BCG-style two-dimensional diagnostic framework that separates them. Classifiers are placed on a 2x2 grid by Spiegelhalter Z-statistic and AUC-ROC expected rank, then assigned to one of four archetypes: Eagle (good on both axes), Bull (strong discrimination, poor calibration), Sloth (well-calibrated, weak discriminator), and Mole (poor on both). Each archetype carries a distinct prescription. We populate the matrix from a large-scale empirical study spanning 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 suite. The assignment is unambiguous. CatBoost, TabICL, EBM, TabPFN, GBC, and Random Forest are Eagles. XGBoost, LightGBM, and HGB are Bulls; Venn-Abers calibration cuts log-loss by 6.5 to 12.6% on Bulls but degrades Eagles by 2.1%. SVM, LR, LDA, and the empirical base-rate predictor are Sloths. MLP, KNN, Naive Bayes, and ExtraTrees are Moles. A theoretical asymmetry follows: no order-preserving post-hoc calibrator can add discriminatory power (Proposition 1), so calibration is the fixable part and discrimination is the hard part. The practical rule is direct: do not optimise aggregate Brier score without first decomposing it; optimise discrimination first, then fix calibration post-hoc. Code and raw experimental data are available at https://github.com/valeman/classifier_calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。