arXiv:2411.17983stat.MEcs.AI2024-11被引 13

提出OptCS框架,让模型优化后仍能可靠筛选重要样本。

Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization

  • 允许在数据驱动下优化模型,不破坏检验有效性
  • 三种方法实现无样本分割建模,提升检测功率
  • 适合药研和医学报告生成等需高精度筛选场景

在共形选择中,模型选择常因标签与无标签数据的可交换性被破坏而面临挑战。现有方法要求模型选择独立于构建共形p值和校准选择集的数据,但在标注数据有限时,希望既能数据驱动地选择最优模型,又能避免样本分割导致的效能损失。本文提出通用框架OptCS,可在大量数据重用和复杂p值依赖情况下,通过新颖的多重检验程序保持有限样本下的错误发现率(FDR)控制,确保统计推断的有效性。我们基于该框架提出三种不同的FDR控制方法:(i)从多个预训练模型中选取最强者;(ii)使用全部数据进行模型拟合,无需样本分割;(iii)结合全样本建模与选择。通过模拟实验和真实应用(药物发现、放射科报告生成中的大语言模型对齐)验证了方法的有效性。

原文摘要 · Abstract (English)

Model selection/optimization in conformal inference is challenging, since it may break the exchangeability between labeled and unlabeled data. We study this problem in the context of conformal selection, which uses conformal p-values to select ``interesting'' instances with large unobserved labels from a pool of unlabeled data, while controlling the FDR in finite sample. For validity, existing solutions require the model choice to be independent of the data used to construct the p-values and calibrate the selection set. However, when presented with many model choices and limited labeled data, it is desirable to (i) select the best model in a data-driven manner, and (ii) mitigate power loss due to sample splitting. This paper presents OptCS, a general framework that allows valid statistical testing (selection) after flexible data-driven model optimization. We introduce general conditions under which OptCS constructs valid conformal p-values despite substantial data reuse and handles complex p-value dependencies to maintain finite-sample FDR control via a novel multiple testing procedure. We instantiate this general recipe to propose three FDR-controlling procedures, each optimizing the models differently: (i) selecting the most powerful one among multiple pre-trained candidate models, (ii) using all data for model fitting without sample splitting, and (iii) combining full-sample model fitting and selection. We demonstrate the efficacy of our methods via simulation studies and real applications in drug discovery and alignment of large language models in radiology report generation.

共形推断FDR控制模型选择医学应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。