Few-shot学习中置信区间计算方法存在误导，新研究提出改进方案
Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning
- 用无放回抽样重新计算置信区间，更真实反映数据随机性
- 现有方法显著低估不确定性，可能导致错误结论
- 推荐使用配对检验并优化任务采样策略，适合方法评估者
Few-shot学习（FSL）中计算置信区间（CI）的主流方法是带放回地抽样任务，这使置信区间仅反映抽样器的随机性，而忽略了数据本身的不确定性。我们通过对比有放回与无放回抽样下的置信区间，发现主流方法存在显著低估。这一现象揭示了现有研究中对置信区间的误读问题。研究证明，使用配对检验可部分缓解该问题，并探索通过有策略地选择特定规模的任务来进一步缩小置信区间。此外，我们引入了一个新的优化基准，可通过 https://github.com/RafLaf/FSL-benchmark-again 获取。
原文摘要 · Abstract (English)
The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。