提出新型变分估计方法，精准衡量模型预测概率与实际频率的偏差。
A Variational Estimator for $L_p$ Calibration Errors
- 基于变分框架，扩展至L_p散度的校准误差估计
- 可区分过度自信与不足自信，避免传统方法高估误差
- 代码开源，支持多分类场景校准评估
校准——确保预测概率与真实类别频率一致——是机器学习系统可靠性的基本要求。传统校准误差评估依赖发散函数，通过预测值与经验频率间期望发散来衡量。在多分类场景中，准确估计该量十分困难。本文将近期一种变分框架拓展至由L_p散度诱导的校准误差，覆盖更广泛的情形。所提方法能分离过度自信与不足自信问题，且相较于非变分方法，不会导致误差高估。我们进行了充分实验，并将代码集成至开源工具包probmetrics（https://github.com/dholzmueller/probmetrics），便于校准误差评估。
原文摘要 · Abstract (English)
Calibration$\unicode{x2014}$the problem of ensuring that predicted probabilities align with observed class frequencies$\unicode{x2014}$is a basic desideratum for reliable prediction with machine learning systems. Calibration error is traditionally assessed via a divergence function, using the expected divergence between predictions and empirical frequencies. Accurately estimating this quantity is challenging, especially in the multiclass setting. Here, we show how to extend a recent variational framework for estimating calibration errors beyond divergences induced induced by proper losses, to cover a broad class of calibration errors induced by $L_p$ divergences. Our method can separate over- and under-confidence and, unlike non-variational approaches, avoids overestimation. We provide extensive experiments and integrate our code in the open-source package probmetrics (https://github.com/dholzmueller/probmetrics) for evaluating calibration errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。