发现校准偏差随参数复杂度上升、校准集增大而下降，可指导实际应用。
Parametric Scaling Law of Tuning Bias in Conformal Prediction
- 通过实证与理论推导，揭示了参数调优偏差的缩放规律。
- 偏差随参数空间复杂度上升，随校准集规模增加而减小。
- 为降低偏差提供理论依据，适合做不确定性量化研究者参考。
置信预测是一种流行的不确定性量化框架，能构建具有覆盖率保证的预测集。为满足交换性假设，许多置信预测方法需要额外的保留集进行参数调优。然而，违反此原则对覆盖率的影响尚未充分探索，导致实际应用中存在模糊性。本文通过实验发现，在多数置信预测方法中，使用相同数据集进行调优与校准所引入的调优偏差（即覆盖率差距）可忽略不计。特别地，我们观察到调优偏差的缩放规律：该偏差随参数空间复杂度增加而上升，随校准集大小增加而下降。我们建立了形式化理论框架，量化调优偏差并严格证明其缩放规律，推导出偏差的上界。最后，基于所建立的理论，讨论了如何有效降低调优偏差。
原文摘要 · Abstract (English)
Conformal prediction is a popular framework of uncertainty quantification that constructs prediction sets with coverage guarantees. To uphold the exchangeability assumption, many conformal prediction methods necessitate an additional holdout set for parameter tuning. Yet, the impact of violating this principle on coverage remains underexplored, making it ambiguous in practical applications. In this work, we empirically find that the tuning bias - the coverage gap introduced by leveraging the same dataset for tuning and calibration, is negligible for simple parameter tuning in many conformal prediction methods. In particular, we observe the scaling law of the tuning bias: this bias increases with parameter space complexity and decreases with calibration set size. Formally, we establish a theoretical framework to quantify the tuning bias and provide rigorous proof for the scaling law of the tuning bias by deriving its upper bound. In the end, we discuss how to reduce the tuning bias, guided by the theories we developed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。