提出可高效估算二分类器校准误差的新方法，无需假设且计算开销小。
Measuring Uncertainty Calibration
- 基于有界变差的校准函数，给出误差上界
- 修改分类器后可快速上界估计，性能影响小
- 适用于真实数据，适合实际部署时评估校准
本文针对从有限数据集估计二分类器的 $L_1$ 校准误差问题做出两项贡献。首先，对校准函数具有有界变差的任意分类器，给出了误差上界。其次，提出一种修改任意分类器的方法，使其校准误差可高效上界估计，且性能损失小、无需严格假设。所有结果均为非渐近且分布无关。最后提供实践中测量校准误差的建议。所提方法可在真实数据集上运行，仅需适度计算开销。
原文摘要 · Abstract (English)
We make two contributions to the problem of estimating the $L_1$ calibration error of a binary classifier from a finite dataset. First, we provide an upper bound for any classifier where the calibration function has bounded variation. Second, we provide a method of modifying any classifier so that its calibration error can be upper bounded efficiently without significantly impacting classifier performance and without any restrictive assumptions. All our results are non-asymptotic and distribution-free. We conclude by providing advice on how to measure calibration error in practice. Our methods yield practical procedures that can be run on real-world datasets with modest overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。