提出鲁棒共形选择框架,应对标注数据含噪时的误选率失控问题。
Robust Conformalized Selection with Noisy Responses
- 通过条件分组将标签噪声转化为可处理的协变量偏移问题
- 在含噪校准数据下仍能严格控制假发现率(FDR)
- 适用于药物发现、大模型对齐等需要高置信度筛选的任务
共形选择已被广泛用于从大规模数据集中筛选高质量候选者,并提供严格的不确定性量化,如可靠标注、药物发现及大语言模型对齐。然而,现有方法假设校准数据中响应值干净,这一假设在实践中极少成立。本文将上述任务建模为选择真实预测标签或响应值超过阈值的候选者。我们证明,在校准数据被污染时,现有共形选择方法无法控制假发现率(FDR),或导致严重功效损失。为此,我们提出鲁棒共形选择(RCS),一种在一般标签污染下仍能保证有效FDR控制的统一选择分类框架。RCS的核心思想是:通过分别对不同类别进行条件化,将难以处理的标签噪声转化为局部协变量偏移问题,进而实现协变量调整的经验贝叶斯型假选择数估计。我们建立了RCS的渐近FDR控制、功效最优性与鲁棒性等统计性质。我们还提出了基于随机响应模型的RCS实例,并将其扩展至选择高响应值候选者的任务。在模拟与真实数据集上的大量实验验证了RCS的有效性。
原文摘要 · Abstract (English)
Conformalized selection has been widely applied to select high-quality candidates from large datasets with rigorous uncertainty quantification, such as reliable labeling, drug discovery, and the alignment of large language models. Nevertheless, existing methods assume clean responses on calibration data, an assumption that rarely holds in practice. In this paper, we formulate the above tasks as selecting candidates with true predicted labels or with responses exceeding certain values. We demonstrate that existing conformal selection methods fail to control the false discovery rate (FDR) or suffer from severe power loss under contaminated calibration data. To that end, we propose Robust Conformalized Selection (RCS), a unified framework for selective classification with valid FDR control under general label contamination. The key insight of RCS lies in a novel statistical reduction: by separately conditioning on different classes, we translate the intractable label noise into a localized covariate shift problem, which then enables a covariate-adjusted empirical-Bayes-type estimate of the number of false selections. Statistical properties such as the asymptotic FDR control, power optimality, and robustness of RCS are established. We further develop an instantiation of RCS under randomized response model, and also apply RCS to the task of selecting candidates with large response values. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of RCS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。