不依赖预测反馈,用查询机制实现高可靠不确定性量化。
Online Conformal Prediction Beyond Feedback

- 通过查询真值而非依赖部署反馈更新预测集
- 覆盖率达β - O(T⁻¹⁄³),仅需查询约T⁻¹⁄³比例的样本
- 适合无法获取部署结果反馈的安全关键场景
在安全关键应用中,不确定性量化至关重要。在线合规定性预测(OCP)通过构建保证在用户指定频率内包含真实标签的预测集,为任意黑箱分类器和非独立同分布数据流提供理论保障的不确定性量化。传统OCP依赖已部署预测的反馈来更新预测集。本文研究无反馈场景下的OCP:每轮学习者可输出预测集或查询真值标签,但不能同时进行。该问题被建模为部分监控博弈,其中预测动作无观测,而查询动作揭示标签。奖励函数设计促使学习者输出小规模预测集的同时确保足够高的覆盖率。为此,我们提出带查询的OCP(OCPQ),基于Cesa-Bianchi等(2004)的标签高效预测器改进。对于任意黑箱分类器和任意长度为T的(非独立同分布)盲观数据流,OCPQ具有期望遗憾O(T²⁄³),期望覆盖率至少为β - O(T⁻¹⁄³),且仅需期望查询比例为T⁻¹⁄³的轮次。该方法在覆盖率上媲美基于强化学习的OCP,却无需依赖部署预测的反馈。真实数据集实验进一步验证了其有效性。
原文摘要 · Abstract (English)
Uncertainty quantification is essential when deploying machine learning models in safety-critical applications. Online conformal prediction (OCP) provides theoretically principled uncertainty quantification for arbitrary black-box classifiers and non-i.i.d. data streams by constructing prediction sets that are guaranteed to contain the true label at a user-specified frequency. OCP usually updates prediction sets using feedback from previously deployed predictions. We instead study an OCP setting beyond feedback: on each round, the learner can either output a prediction set or query the correct label, but not both. Thus, no deployed prediction is ever evaluated directly. We reduce this problem to a partial monitoring game in which prediction actions return no observation and a separate query action reveals the label. The reward function is constructed in a way that encourages the learner to output small prediction sets while ensuring that the correct label is covered with a sufficiently high probability. To solve this game, we develop OCP with queries (OCPQ) by adapting the label efficient forecaster of Cesa-Bianchi, Lugosi, and Stoltz (2004) to our setting. For any black box classifier and any (non-i.i.d.) oblivious data stream of length $T$, OCPQ has $O(T^{2/3})$ expected regret and expected coverage at least $β-O(T^{-1/3})$ for a user-defined $β$, while querying only an expected $T^{-1/3}$ fraction of rounds. This provides coverage comparable to bandit-based OCP methods while requiring no feedback from deployed prediction sets. Experiments on real-world datasets further demonstrate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。