解释模型校准的原理与常用评估方法，揭示经典指标的局限性。
Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)
- 用期望校准误差（ECE）衡量模型置信度与真实准确率的一致性
- 发现ECE在某些情况下会低估校准问题，导致误判
- 适合想理解校准概念和评估缺陷的研究新手
一个可靠的模型必须具备良好的校准能力，即其预测置信度应与实际正确率一致。本文介绍最常用的校准定义及评估指标——期望校准误差（Expected Calibration Error, ECE）。通过可视化分析，揭示ECE在不同数据分布下的局限性，指出其可能无法有效反映模型的真实校准水平。文章还讨论了由此引发的对新校准概念与评估方法的需求，强调虽有多种改进方案，但ECE仍被广泛使用。本文不深入探讨校准方法或全面综述相关研究，而是旨在为初学者提供校准概念与评估工具的直观理解，同时提醒注意当前主流评估指标的潜在缺陷。
原文摘要 · Abstract (English)
To be considered reliable, a model must be calibrated so that its confidence in each decision closely reflects its true outcome. In this blogpost we'll take a look at the most commonly used definition for calibration and then dive into a frequently used evaluation measure for model calibration. We'll then cover some of the drawbacks of this measure and how these surfaced the need for additional notions of calibration, which require their own new evaluation measures. This post is not intended to be an in-depth dissection of all works on calibration, nor does it focus on how to calibrate models. Instead, it is meant to provide a gentle introduction to the different notions and their evaluation measures as well as to re-highlight some issues with a measure that is still widely used to evaluate calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。