利用模型预测降低标签成本,更准更快地估计风险
Prediction-Powered Active Testing
- 用预测残差做控制变量,既保持无偏又降低方差
- 在少样本下风险估计误差更小,置信区间更窄
- 适合标签昂贵的场景,如医疗或工业检测
主动测试通过自适应选择需标注的测试点,实现标签高效的風險估計。然而,現有估計器未能利用強大黑箱模型的預測信息,儘管這些預測在標籤仍昂貴的場景中日益可得。為解決此問題,我們提出預測驅動主動測試(PPAT),一種新的標籤高效風險估計框架,結合無偏的LURE估計器與預測驅動的控制變量。與使用有偏偽標籤的代理預測不同,PPAT利用預測進行損失殘差化,保持無偏性同時降低方差。除估計器本身外,PPAT還改變了應採集點的選擇策略:我們推導出針對降低估計器方差的神諭與實用的近似採集規則。此外,我們建立了PPAT的漸近正態性,從而獲得漸近有效的置信區間,實現對估計不確定性的合理量化。在表格回歸和圖像分類任務中,PPAT在風險估計上優於現有方法,其置信區間在更少標籤下達到目標覆蓋率且區間更窄。
原文摘要 · Abstract (English)
Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive. To address this, we propose \textbf{Prediction--Powered Active Testing (PPAT)}, a novel label--efficient risk estimation framework that combines the unbiased LURE estimator \citep{farquhar2021statistical} with a prediction--powered control variate. Rather than using proxy predictions as biased pseudo--labels, PPAT uses them to residualise the loss, preserving unbiasedness while reducing variance. Beyond the estimator itself, PPAT also changes which points should be acquired: we derive oracle and practical surrogate--based acquisition rules tailored to reducing the variance of our estimator. Moreover, we establish asymptotic normality for PPAT, yielding asymptotically valid confidence intervals and thus a principled estimate of the uncertainty around our estimates. Across tabular regression and image--classification tasks, PPAT outperforms existing methods in risk estimation, while its confidence intervals attain the target coverage with substantially fewer labels and smaller widths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。