提出新方法,在类别不平衡且存在未知类别的场景下,仍能给出可靠预测集。
Conformal Inference for Open-Set and Imbalanced Classification
- 设计新型校准p值,检测新样本是否属于未见类别
- 在无限可能类别下仍保持准确覆盖率,极端不平衡时效果更优
- 适合处理开放集分类与数据严重不均衡的场景
本文提出一种用于高度不平衡和开放集分类的合规预测方法,其中存在大量可能类别且训练数据中未必涵盖全部。现有方法需已知有限标签空间,并依赖随机样本分割,当各类别样本数量不足时表现不佳,导致两个问题:(i) 测试时遇到新类别无法提供充分覆盖;(ii) 对已见类别的预测过于保守。为解决此问题,我们引入一类新的合规p值,可检验新样本是否属于未见类别。理论上分析其最优性,并揭示其与经典Good--Turing估计器(用于估算新物种出现概率)之间的深刻关联。为进一步提升不平衡数据的利用效率,我们提出选择性样本分割算法,按标签频率划分训练与校准数据,虽破坏交换性,但通过适当重加权仍可保持有限样本保证。在模拟与真实数据上验证,该方法在无限可能类别、极端类别不平衡的开放集场景中,仍能实现有效覆盖率,并生成更具信息量的预测集。
原文摘要 · Abstract (English)
This paper presents a conformal prediction method for classification in highly imbalanced and open-set settings, where there are many possible classes and not all may be represented in the data. Existing approaches require a finite, known label space and typically involve random sample splitting, which works well when there is a sufficient number of observations from each class. Consequently, they have two limitations: (i) they fail to provide adequate coverage when encountering new labels at test time, and (ii) they may become overly conservative when predicting previously seen labels. To obtain valid prediction sets in the presence of unseen labels, we compute and integrate into our predictions a new family of conformal p-values that can test whether a new data point belongs to a previously unseen class. We study these p-values theoretically, establishing their optimality, and uncover an intriguing connection with the classical Good--Turing estimator for the probability of observing a new species. To make more efficient use of imbalanced data, we also develop a selective sample splitting algorithm that partitions training and calibration data based on label frequency, leading to more informative predictions. Despite breaking exchangeability, this allows maintaining finite-sample guarantees through suitable re-weighting. With both simulated and real data, we demonstrate our method leads to prediction sets with valid coverage even in challenging open-set scenarios with infinite numbers of possible labels, and produces more informative predictions under extreme class imbalance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。