arXiv:2410.09894cs.LG2024-10

在数据少时,不同误差度量对预测置信区间的有效性影响大。

Inductive Conformal Prediction under Data Scarcity: Exploring the Impacts of Nonconformity Measures

  • 用绝对误差、归一化误差和分位数三种方式衡量数据差异
  • 数据量小且噪声高时,区间效率差异明显,无最优度量
  • 数据多未必更准,选对度量比凑数据更重要

置信预测不依赖数据分布假设,是实际应用中可靠的不确定性量化方法。其核心是非符合度量,用于衡量测试样本与训练数据的差异,预测区间的有效性高度依赖所选度量。然而,这一选择在数据稀缺情况下的影响尚未被充分研究。本研究旨在评估绝对误差、归一化绝对误差和分位数三类非符合度量在归纳式置信预测中的表现,重点关注小数据集场景。基于合成与真实数据,我们分析了数据集大小、噪声水平和维度等特征对置信区间效率的影响。结果表明,虽存在差异,但无单一度量始终最优,每种度量的效果均受数据特性显著影响。此外,增加数据量并不总能提升效率,凸显模型调优与度量选择的重要性。

原文摘要 · Abstract (English)

Conformal prediction, which makes no distributional assumptions about the data, has emerged as a powerful and reliable approach to uncertainty quantification in practical applications. The nonconformity measure used in conformal prediction quantifies how a test sample differs from the training data and the effectiveness of a conformal prediction interval may depend heavily on the precise measure employed. The impact of this choice has, however, not been widely explored, especially when dealing with limited amounts of data. The primary objective of this study is to evaluate the performance of various nonconformity measures (absolute error-based, normalized absolute error-based, and quantile-based measures) in terms of validity and efficiency when used in inductive conformal prediction. The focus is on small datasets, which is still a common setting in many real-world applications. Using synthetic and real-world data, we assess how different characteristics -- such as dataset size, noise, and dimensionality -- can affect the efficiency of conformal prediction intervals. Our results show that although there are differences, no single nonconformity measure consistently outperforms the others, as the effectiveness of each nonconformity measure is heavily influenced by the specific nature of the data. Additionally, we found that increasing dataset size does not always improve efficiency, suggesting the importance of fine-tuning models and, again, the need to carefully select the nonconformity measure for different applications.

置信预测不确定性量化小样本非符合度量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。