提出新方法诊断预测系统在局部条件下的可靠性,解决传统评估方式效率低的问题。
Conditional Coverage Diagnostics for Conformal Prediction
- 将条件覆盖率估计转化为分类问题,用风险差衡量偏差
- 实验显示现代分类器比传统方法统计功效高得多
- 适合研究者和工程师调试模型的局部可靠性
评估条件覆盖率仍是判断预测系统可靠性的主要挑战。尽管分位数方法能保证边际覆盖率,但无法确保条件覆盖率正确,导致实践者难以解释局部偏差。为克服现有度量方法样本效率低和过拟合问题,我们将条件覆盖率估计建模为分类问题:当且仅当存在分类器风险低于目标覆盖率时,条件覆盖率被违反。通过选择合适的损失函数,所得风险差可保守估计自然误覆盖度量(如L1、L2距离),甚至能区分过覆盖与欠覆盖,并处理非恒定目标覆盖率。我们称该度量族为目标覆盖率的超额风险(ERT)。实验表明,使用现代分类器比基于简单分类器的已有度量(如CovGap)具有更高的统计功效。我们还利用该度量对多种分位数预测方法进行基准测试。最后,我们开源了ERT及相关度量工具包。这些贡献为理解、诊断和改进预测系统的条件可靠性提供了新视角。
原文摘要 · Abstract (English)
Evaluating conditional coverage remains one of the most persistent challenges in assessing the reliability of predictive systems. Although conformal methods can give guarantees on marginal coverage, no method can guarantee to produce sets with correct conditional coverage, leaving practitioners without a clear way to interpret local deviations. To overcome sample-inefficiency and overfitting issues of existing metrics, we cast conditional coverage estimation as a classification problem. Conditional coverage is violated if and only if some classifier can achieve lower risk than the target coverage. Through the choice of a (proper) loss function, the resulting risk difference gives a conservative estimate of natural miscoverage measures such as L1 and L2 distance, and can even separate the effects of over- and under-coverage, and non-constant target coverages. We call the resulting family of metrics excess risk of the target coverage (ERT). We show experimentally that the use of modern classifiers provides much higher statistical power than simple classifiers underlying established metrics like CovGap. Additionally, we use our metric to benchmark different conformal prediction methods. Finally, we release an open-source package for ERT as well as previous conditional coverage metrics. Together, these contributions provide a new lens for understanding, diagnosing, and improving the conditional reliability of predictive systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。