提出阈值设定下的有效样本量新算法,解决聚类数据导致的覆盖失效问题。
The Exceedance Design Effect: Effective Sample Size for Thresholds under Clustering
- 基于簇内评分同侧概率推导阈值有效样本量
- 发现不同阈值对应不同有效样本量,原方法有偏差
- 适用于需严格保证覆盖率的部署系统,如安全过滤器
许多机器学习系统在校准集的分位数上设定阈值:例如,保真预测器通过校准集第90百分位设置截止点以保证90%覆盖,拒绝门控在模型得分低于第10百分位时拒答,安全过滤器则屏蔽超过参考集第99百分位的输出。这些方法承诺在新数据上仍保持指定覆盖率,但前提是校准样本独立。然而现代流程中样本常共享提示、文档或推理轨迹,存在相关性。统计学自1965年起已知如何通过独立观测等价数调整相关数据,但仅限于均值。本文揭示阈值需不同计算方式:有效样本量取决于簇内评分落在阈值同侧的频率,且随阈值位置变化;数值相似性无关。我们推导出闭式表达式,给出实际部署覆盖率的分布。三个结论:现有符合性文献中的修正量错误,可能高估或低估;数据集无单一有效样本量,每阈值对应一个;误差在多次平均覆盖中不可见,单次部署者将完全承担损失。在25,028个示例的公开校准集上,实测可靠性约1,300个。
原文摘要 · Abstract (English)
Many machine-learning systems set a threshold at a quantile of a calibration set: conformal predictors that promise 90% coverage by drawing their cutoff at the calibration set's 90th percentile, abstention gates that decline to answer when a model's score falls below the calibration set's tenth percentile, safety filters that block any output scoring above the 99th percentile of a reference set. All of them promise that the threshold will hold at the stated rate on new data. The promise assumes the calibration examples are independent, and in modern pipelines they usually are not: they share a prompt, a document, a reasoning trace. Survey statistics has known how to discount correlated data since 1965, by counting how many independent observations a sample is worth, but only for averages. We show that a threshold needs a different count. The count depends on how often clustered scores land on the same side of the threshold, and that changes with where the threshold is set. How similar the scores are as numbers does not enter. We prove a closed-form law for the resulting effective sample size and for the spread of the coverage a deployed system actually sees. Three consequences follow. The correction now used in the conformal literature is the wrong quantity, and can miss in either direction. A dataset has no single effective sample size. It has one for each level the threshold is set at. And the damage is invisible in coverage averaged over many runs, and fully felt by whoever deploys once. On a released calibration set of 25,028 examples, we measure the reliability of about 1,300.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。