对比13种误差置信区间方法，给出可靠推荐
Constructing Confidence Intervals for 'the' Generalization Error -- a Comprehensive Benchmark Study
- 系统比较13种基于重采样与方差估计的置信区间方法
- 在19个任务上验证，发现部分方法覆盖更准且区间更窄
- 适合模型评估者参考，尤其关注结果可信度的研究者
在机器学习中，泛化误差的置信区间（CI）是衡量预测性能的关键工具。尽管已有多种计算方法，但尚无共识说明何种组合最可靠或如何比较。本文首次开展大规模实证研究，系统评估13种CI方法在19个表格回归与分类任务上的表现，使用7种学习器和8种损失函数。我们梳理了方法原理与挑战，统一框架下回顾所有13种方法，并从覆盖率、区间宽度和运行时间三方面评估。研究识别出若干推荐方法。数据集已发布于OpenML，代码开源，支持后续研究。
原文摘要 · Abstract (English)
When assessing the quality of prediction models in machine learning, confidence intervals (CIs) for the generalization error, which measures predictive performance, are a crucial tool. Luckily, there exist many methods for computing such CIs and new promising approaches are continuously being proposed. Typically, these methods combine various resampling procedures, most popular among them cross-validation and bootstrapping, with different variance estimation techniques. Unfortunately, however, there is currently no consensus on when any of these combinations may be most reliably employed and how they generally compare. In this work, we conduct a large-scale study comparing CIs for the generalization error, the first one of such size, where we empirically evaluate 13 different CI methods on a total of 19 tabular regression and classification problems, using seven different inducers and a total of eight loss functions. We give an overview of the methodological foundations and inherent challenges of constructing CIs for the generalization error and provide a concise review of all 13 methods in a unified framework. Finally, the CI methods are evaluated in terms of their relative coverage frequency, width, and runtime. Based on these findings, we can identify a subset of methods that we would recommend. We also publish the datasets as a benchmarking suite on OpenML and our code on GitHub to serve as a basis for further studies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。