提出新方法确保分类预测集在长尾数据下对每类平均覆盖,兼顾公平性与实用性。
Conformal Prediction with Macro-Coverage Guarantees
- 用标签加权的置信推断方法,实现有限样本下的宏观覆盖率保证。
- 在两个大规模图像数据集上验证,新方法在稀有类别上仍保持稳定覆盖。
- 适合长尾分布场景,尤其关注少数类性能的研究者值得关注。
预测集需具备高覆盖率才具实用价值,但不同覆盖率定义的实用性各异。分类任务中,类条件覆盖率要求每个类别均达到目标准确率,但在类别稀疏且校准样本少时难以满足;而边际覆盖率仅要求整体平均覆盖,易忽略低概率类别。为调和二者,近年提出宏观覆盖率,即各分类别条件覆盖率的无权重平均值,特别适用于长尾分布。本文证明,标签加权的置信推断可实现有限样本下宏观覆盖率的严格保证,并推广至任意类别分组加权平均的广义宏观覆盖率目标。进一步刻画了满足给定目标的最小预测集形式,并提出相应置信评分函数。理论结果在两个大规模图像分类数据集上得到验证。
原文摘要 · Abstract (English)
Prediction sets should have high coverage to be useful, but some coverage notions are more practically relevant than others. In the classification setting, class-conditional coverage requires that the prediction set (i.e., the set of candidate labels for a new test point) must achieve the target accuracy level within each class, which may be challenging to satisfy when many classes are rare and have few calibration points. At the other extreme, marginal coverage requires only that coverage holds on average over the distribution of all classes, which can lead to low-probability labels being essentially ignored. To find a middle ground, recent work has introduced macro-coverage, defined as the unweighted average of class-conditional coverages. Macro-coverage offers a compromise between marginal coverage and class-conditional coverage that is particularly appropriate for long-tailed settings. In this work, we show that label-weighted conformal prediction can be used to produce prediction sets with a finite-sample macro-coverage guarantee, and more generally a guarantee on a family of generalized macro-coverage objectives that aggregate coverage at the level of arbitrary class groupings and take a weighted average. We further characterize the form of the smallest prediction sets satisfying a given generalized macro-coverage objective and propose a corresponding conformal score function. We validate our theoretical results on two large-scale image classification datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。