arXiv:2603.14092cs.LGstat.ME2026-03

提出新校准度量SMECE,专用于概率标签场景

Soft Mean Expected Calibration Error (SMECE): A Calibration Metric for Probabilistic Labels

  • 用标签均值替换eCE中的硬标签比例,更适配概率标签
  • 在非二元概率标签下,传统eCE会系统性偏差,SMECE可纠正
  • 兼容原有方法,适合知识蒸馏、生成模型等场景

期望校准误差(ECE)是机器学习中主流的校准度量,它将预测概率与二元结果的实际频率进行比较。然而,在许多现代场景中,标签本身也是概率:如放射科医生的置信度、知识蒸馏中教师模型的软输出、生成模型推导出的类别后验,或标注者一致率。这些情况下,使用ECE会造成类别错误——它将标签的概率信息强制转换为二元比较,从而丢失关键信息。这并非数据噪声导致的近似偏差,而是结构性错配,样本量越大,其误差反而越收敛于错误答案。本文提出软均值期望校准误差(SMECE),专用于概率标签场景。对ECE公式仅做一行修改:将每个预测桶内的经验硬标签比例替换为该桶样本的平均概率标签。当标签为二元时,SMECE精确退化为ECE,因此是严格推广。

原文摘要 · Abstract (English)

The Expected Calibration Error (ece), the dominant calibration metric in machine learning, compares predicted probabilities against empirical frequencies of binary outcomes. This is appropriate when labels are binary events. However, many modern settings produce labels that are themselves probabilities rather than binary outcomes: a radiologist's stated confidence, a teacher model's soft output in knowledge distillation, a class posterior derived from a generative model, or an annotator agreement fraction. In these settings, ece commits a category error - it discards the probabilistic information in the label by forcing it into a binary comparison. The result is not a noisy approximation that more data will correct. It is a structural misalignment that persists and converges to the wrong answer with increasing precision as sample size grows. We introduce the Soft Mean Expected Calibration Error (smece), a calibration metric for settings where labels are of probabilistic nature. The modification to the ece formula is one line: replace the empirical hard-label fraction in each prediction bin with the mean probability label of the samples in that bin. smece reduces exactly to ece when labels are binary, making it a strict generalisation.

校准误差概率标签知识蒸馏生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。