用少量真实标签+合成标签实时监测模型风险，发现有害分布偏移。
Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
- 结合真实与合成标签，构建随时有效的风险下界
- 在图像、大模型和通信任务中检测到有害分布偏移
- 无需假设，有限样本下保证误报率不超阈值
我们研究动态环境中标签数据稀缺时的模型性能监控问题。为此，提出预测驱动的风险监控（PPRM），一种基于预测驱动推断（PPI）的半监督风险监控方法。PPRM通过融合少量真实标签与合成标签，构建任意时刻有效的风险下界。通过与名义风险上界进行阈值比较，检测有害分布偏移，并在不依赖假设的前提下，满足类型一误差的有限样本保证。我们在图像分类、大语言模型（LLM）及电信监控任务中进行了广泛实验，验证了PPRM的有效性。
原文摘要 · Abstract (English)
We study the problem of monitoring model performance in dynamic environments where labeled data are limited. To this end, we propose prediction-powered risk monitoring (PPRM), a semi-supervised risk-monitoring approach based on prediction-powered inference (PPI). PPRM constructs anytime-valid lower bounds on the running risk by combining synthetic labels with a small set of true labels. Harmful shifts are detected via a threshold-based comparison with an upper bound on the nominal risk, satisfying assumption-free finite-sample guarantees on the type-I error. We demonstrate the effectiveness of PPRM through extensive experiments on image classification, large language model (LLM), and telecommunications monitoring tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。