提升预测集在高置信度错误预测时的覆盖率
Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores
- 基于置信度和非参数信任分数构建条件覆盖优化的预测算法
- 在多个图像数据集上显著改善了类别、子群体及人口统计组的覆盖率
- 特别强化了对模型过度自信但错误预测场景的覆盖能力
标准共形预测仅提供边际覆盖率保证，而真正有用的预测集应确保在每个测试点上的条件覆盖率。然而，在有限样本下无法实现精确的、分布无关的条件覆盖率。本文提出一种替代共形预测算法，聚焦于模型对错误预测过度自信的场景。通过分析边际有效共形预测中的误覆盖事件，发现误覆盖率随分类器置信度及其与贝叶斯最优分类器的偏差而变化。受此启发，我们开发了一种新方法，将条件覆盖率目标限定于两个变量：分类器的置信度与衡量其偏离贝叶斯分类器程度的非参数信任分数。在多个图像数据集上的实证评估表明，该方法相较于标准共形预测显著提升了条件覆盖率，包括类别条件覆盖率、任意子群体覆盖率以及人口统计群体覆盖率。
原文摘要 · Abstract (English)
Standard conformal prediction offers a marginal guarantee on coverage, but for prediction sets to be truly useful, they should ideally ensure coverage conditional on each test point. Unfortunately, it is impossible to achieve exact, distribution-free conditional coverage in finite samples. In this work, we propose an alternative conformal prediction algorithm that targets coverage where it matters most--in instances where a classifier is overconfident in its incorrect predictions. We start by dissecting miscoverage events in marginally-valid conformal prediction, and show that miscoverage rates vary based on the classifier's confidence and its deviation from the Bayes optimal classifier. Motivated by this insight, we develop a variant of conformal prediction that targets coverage conditional on a reduced set of two variables: the classifier's confidence in a prediction and a nonparametric trust score that measures its deviation from the Bayes classifier. Empirical evaluation on multiple image datasets shows that our method generally improves conditional coverage properties compared to standard conformal prediction, including class-conditional coverage, coverage over arbitrary subgroups, and coverage over demographic groups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。