通过聚类学习特征空间,为不同子群体分别校准概率,提升可靠性。
Clustered Calibration: Representation-Aware Probability Calibration via Learned Subpopulations
- 在学习到的特征空间中聚类,识别出不同子群体进行针对性校准。
- 在六大数据集上优于或匹配现有方法,负对数似然和Brier得分更优。
- 适合高风险场景,如医疗、自动驾驶,需精准概率估计的领域。
确保预测概率与实际发生频率一致,在临床决策支持、自动驾驶和金融风险评估等高风险领域至关重要。现有校准方法通常采用单一全局变换或基于预测置信度的后处理分箱,难以利用子群体间的可靠性差异。本文提出簇化校准(Clustered Calibration),通过在学习到的特征空间(如覆盖率向量、SHAP值、CNN激活值、Transformer嵌入)中聚类,识别子群体,并在层次收缩约束下拟合各簇专属的参数化校准器,实现上下文感知的校准同时保持全局稳定性。在六个表格数据集及额外的图像和文本基准上,该方法在负对数似然和Brier分数上持续优于或匹配强基线,同时保持AUC与准确率。我们进一步通过分析与实证表明,固定分箱的期望校准误差(ECE)可能错误排序软性、区域感知校准器,即使在良好评分规则下亦然,因此主张以对数损失和Brier分数作为此类场景下的模型选择更可靠依据。
原文摘要 · Abstract (English)
Ensuring that predicted probabilities align with observed frequencies is critical in high-stakes domains such as clinical decision support, autonomous driving and financial risk assessment. Existing calibration methods typically apply a single global transformation or rely on post-hoc binning over predicted confidences, limiting their ability to exploit heterogeneous reliability across sub-populations. We propose Clustered Calibration, a representation-aware framework that identifies sub-populations via clustering in learned feature spaces (e.g., coverage vectors, SHAP values, CNN activations, Transformer embeddings) and fits a soft mixture of cluster-specific parametric calibrators under hierarchical shrinkage toward a global mapping. This design yields context-specific calibration while maintaining global stability. Across six tabular datasets and additional image and text benchmarks, clustered calibration consistently improves or matches strong global calibrators in terms of negative log-likelihood and Brier score, while preserving AUC and accuracy. We further show, both analytically and empirically, that fixed-bin Expected Calibration Error (ECE) can mis-rank soft, region-aware calibrators even when proper scoring rules improve, and we advocate for log-loss and Brier as more reliable bases for model selection in such settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。