小样本医学图像分类中，交叉验证比传统验证方法更可靠地预测模型真实性能。
On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers
- 采用交叉验证替代固定或重排验证，提升超参数优化时对测试性能的估计准确性。
- 在小样本医疗图像数据上，交叉验证使性能估计误差降低40%以上，且随样本量增加效果减弱。
- 适合资源充足的小样本医学影像研究者，尤其关注模型泛化性能可靠性时使用。
超参数优化（HPO）显著影响深度学习图像分类器性能，但针对小样本场景（如医学影像）如何获取可靠的验证信号仍缺乏实证指导。本文比较了三种HPO协议：固定验证集（F）、重排验证集（R）和五折交叉验证（C），以绝对性能估计误差（AEE，即最优配置在验证集与测试集上的AUROC差值）为评估指标。实验在三个公开数据集上进行：两个二分类医学影像数据集（RSNA肺炎胸片、去噪HAM10000皮肤病变）和200类自然图像数据集（Tiny ImageNet），涵盖不同开发集规模 $n$ 及两种主干网络（所有数据集用ResNet-18，RSNA用ViT-S/16）。结果表明，在医学数据集上，交叉验证在所有样本量下均优于两种固定验证方式，尤其在小样本时优势最大，误差降幅达40%以上；在Tiny ImageNet上三者表现相近，误差均接近零。测试AUROC整体差异不大。固定验证的平均AEE在11/12个医学任务中优于重排验证，但支持度不一致。结论：当计算资源允许时，推荐在小样本医学图像分类中使用交叉验证进行超参数优化，以获得更可靠的测试性能预估。
原文摘要 · Abstract (English)
Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of {\em absolute performance-estimation error} (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all datasets, Vision Transformer (ViT-S/16) on RSNA). On the medical datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。