对比两种校准方法在不同条件下的稳定性，发现温度缩放更鲁棒。
Condition-Stratified Robustness Analysis of Post-Hoc Calibration Methods for Probabilistic Classifiers
- 按四种控制条件分层评估校准方法性能
- 温度缩放在所有条件下误差更小，校准斜率更接近理想值
- 结果依赖具体指标和条件，不保证外部可迁移
后训练校准广泛用于修正分类器的概率输出，但多数评估仅报告整体性能，未检验同一数据集内不同运行条件下的表现是否一致。本文开展预注册的条件分层稳健性分析，比较温度缩放（TEMP）与等单调回归（ISO）在四个受控条件（C1–C4）下的表现。检验四组假设：判别力差异（带霍尔姆校正）、布里尔分数差异、校准斜率结果，以及最优条件下的AUROC差异。TEMP减去ISO的判别力差异在各条件下均很小（-0.0155至0.0139），霍尔姆校正p值均为0.9895。TEMP的布里尔分数差异始终为负（C1: -0.0002 到 C4: -0.0074），而ISO出现符号反转。TEMP校准斜率在各条件中更接近单位值（范围0.7597–0.9493），优于ISO（0.1364–0.2726）。AUROC差异从C1的近零（-0.0004）变为C4的正值（0.0264）。结果表明，数据集内稳健性取决于条件与评价指标，不作外部可迁移性主张。
原文摘要 · Abstract (English)
Post-hoc calibration is widely adopted to correct probability estimates from trained classifiers, yet most evaluations report aggregate performance without testing whether that performance holds across distinct operating conditions within a single dataset. We present a pre-registered, condition-stratified robustness analysis comparing temperature scaling (TEMP) and isotonic regression (ISO) across four controlled conditions (C1--C4). Four hypothesis groups are evaluated: discrimination deltas with Holm-corrected multiplicity control (H1), Brier score differences (H2), calibration slope outcomes (H3), and AUROC differences under best-condition setups (H4). TEMP-minus-ISO discrimination deltas remain small across all conditions (-0.0155 to 0.0139), with Holm-adjusted p-values of 0.9895 everywhere. TEMP Brier differences are consistently negative (C1: -0.0002 through C4: -0.0074), while ISO shows sign reversals. TEMP calibration slopes stay closer to unity in every condition (range 0.7597--0.9493) than ISO slopes (0.1364--0.2726). AUROC differences shift from near zero in C1 (-0.0004) to positive in C4 (0.0264). These results establish that in-dataset robustness is condition-dependent and metric-specific. No claim of external transportability is made.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。