解决标注模糊时模型置信度校准失效问题,无需重新训练即可提升可靠性。
Confidence Calibration under Ambiguous Ground Truth
- 提出基于完整标注分布的校准方法,避免传统多数投票带来的偏差。
- 在多个数据集上,新方法使真实标签误差降低55%-87%。
- 仅需模型自身置信度即可实现良好校准,适合无标注者数据场景。
置信度校准通常假设每个输入对应唯一真实标签,但在标注者存在分歧的真实场景下该假设失效。基于多数投票标签的传统校准方法在常规评估中看似准确,实则与标注者分布存在显著偏差。我们证明这种失败具有结构性:在简化假设下,温度缩放倾向于低估标注不确定性,且真实标签校准误差随标注熵增加而单调上升。为此,我们提出一系列面向模糊标注的后处理校准器,优化针对完整标签分布的合理评分规则,无需模型重训练。方法涵盖不同标注需求:Dirichlet-Soft利用全标注分布,在各设置下表现最优;单标注样本下的蒙特卡洛温度缩放(MCTS S=1)在所有基准上达到与全分布校准相当效果,表明预聚合标签分布非必需;标签平滑温度缩放(LS-TS)仅依赖投票标签,通过模型自身置信度构建数据驱动伪软目标。在四个包含真实多标注者分布(CIFAR-10H, ChaosNLI)和临床启发合成标注(ISIC~2019, DermaMNIST)的数据集上,Dirichlet-Soft相较温度缩放将真实标签ECE降低55%-87%,而LS-TS在无需标注者数据情况下使ECE降低9%-77%。
原文摘要 · Abstract (English)
Confidence calibration assumes a unique ground-truth label per input, yet this assumption fails wherever annotators genuinely disagree. Post-hoc calibrators fitted on majority-voted labels, the standard single-label targets used in practice, can appear well-calibrated under conventional evaluation yet remain substantially miscalibrated against the underlying annotator distribution. We show that this failure is structural: under simplifying assumptions, Temperature Scaling is biased toward temperatures that underestimate annotator uncertainty, with true-label miscalibration increasing monotonically with annotation entropy. To address this, we develop a family of ambiguity-aware post-hoc calibrators that optimise proper scoring rules against the full label distribution and require no model retraining. Our methods span progressively weaker annotation requirements: Dirichlet-Soft leverages the full annotator distribution and achieves the best overall calibration quality across settings; Monte Carlo Temperature Scaling with a single annotation per example (MCTS S=1) matches full-distribution calibration across all benchmarks, demonstrating that pre-aggregated label distributions are unnecessary; and Label-Smooth Temperature Scaling (LS-TS) operates with voted labels alone by constructing data-driven pseudo-soft targets from the model's own confidence. Experiments on four benchmarks with real multi-annotator distributions (CIFAR-10H, ChaosNLI) and clinically-informed synthetic annotations (ISIC~2019, DermaMNIST) show that Dirichlet-Soft reduces true-label ECE by 55-87% relative to Temperature Scaling, while LS-TS reduces ECE by 9-77% without any annotator data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。