arXiv:2607.13423cs.LG2026-07

温度校准在人类标签有歧义时失效,模型越大差距越明显。

Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions

  • 用人类标注的软标签测试温度校准效果,发现其不如直接针对软标签校准
  • 视觉模型误差平均0.003,语言模型平均0.079,模型越大差距越大
  • 适用于高风险场景部署,关注标签不确定性的研究者必读

温度校准是现代深度学习中主流的后处理校准方法,其理论基础依赖于一个很少明说的假设:真实标签为确定性独热编码。然而实践中,标签常为软标签、众包或分布式,反映人类标注者的真实分歧而非噪声。本文在两个公开数据集CIFAR-10H和ChaosNLI上,使用三种模型规模,在硬标签与软标签目标下进行评估。九种配置中均发现正向软标签校准差距:基于硬标签校准的模型在软标签上表现持续劣于理想校准模型,Brier Score差距为0.002至0.134。视觉领域差距随模型规模单调上升,语言领域平均差距(0.079)远大于视觉(0.003)。在MNLI衍生子集上虽出现规模排序反转,但因接近随机准确率,视为不具结论性。作为另一基准,多类等距回归也显示所有配置均有正向软标签差距,且语言领域更大。结果表明,基于多数投票标签的校准协议在标签存在结构性歧义时系统性低估模型可靠性,对安全关键场景部署有直接影响。

原文摘要 · Abstract (English)

Temperature scaling is the dominant post-hoc calibration method in modern deep learning. Its theoretical justification rests on an assumption that is rarely stated explicitly: that ground-truth labels are one-hot and deterministic. In practice, labels are frequently soft, crowd-sourced, or genuinely distributional, reflecting real disagreement among human annotators rather than annotation noise. We study whether temperature scaling retains its calibration properties when this assumption is violated, and whether any resulting degradation depends on model scale. Using CIFAR-10H and ChaosNLI, two publicly available datasets with human-annotated soft label distributions, we evaluate three model scales per modality under both hard one-hot and soft distributional label targets. Across all nine configurations we find a positive soft-label calibration gap: temperature scaling calibrated on hard labels consistently underperforms an oracle calibrated directly on soft labels, with Brier Score gaps ranging from 0.002 to 0.134. The gap grows monotonically with model scale in the vision domain and on the SNLI-derived split of ChaosNLI, and is substantially larger in the language domain (mean gap 0.079) than in vision (mean gap 0.003). A scale-ordering reversal on the MNLI-derived split remains after matched-domain training; we treat it as inconclusive for the scale hypothesis and attribute it primarily to near-chance accuracy on that split. As a second post-hoc baseline, multiclass isotonic regression yields the same qualitative conclusion: positive soft-label gaps in all nine configurations, and larger gaps in language than in vision. These findings suggest that calibration protocols built on majority-vote labels systematically misstate model reliability wherever label ambiguity is structural, with direct consequences for deployment in safety-critical settings.

模型校准软标签语言模型视觉模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。