用超级计算提升医疗影像模型评估的可靠性与可复现性
Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models
- 嵌套交叉验证+自动调参+高性能计算协同，量化测试性能波动
- 在胸部X光和OCT数据集上显著降低评估结果方差
- 适合医疗AI研发者构建可信、可部署的深度学习模型
深度学习模型在医学影像中的真实性能评估存在变异性和偏差，现有单次固定测试集方法无法量化性能估计的方差。本研究提出NACHOS（嵌套自动交叉验证与超参数优化的超级计算框架），整合嵌套交叉验证（NCV）与自动化超参数优化（AHPO），在并行化高性能计算（HPC）环境下运行。在胸部X光库和光学相干断层扫描（OCT）数据集上，采用多种数据划分策略进行验证。除性能评估外，还提出DACHOS框架，利用交叉验证与超参数优化，在全数据集上构建最终模型，提升部署预期性能。结果表明，NCV有助于量化并减少评估方差，AHPO能跨测试折一致优化超参数，而HPC确保计算可行性。通过集成这些方法，NACHOS与DACHOS提供了一个可扩展、可复现且可信的深度学习模型评估与部署框架。为促进公开可用性，完整开源代码已发布于https://github.com/thepanlab/NACHOS。
原文摘要 · Abstract (English)
Background and Objectives: The variability and biases in the real-world performance benchmarking of deep learning models for medical imaging compromise their trustworthiness for real-world deployment. The common approach of holding out a single fixed test set fails to quantify the variance in the estimation of test performance metrics. This study introduces NACHOS (Nested and Automated Cross-validation and Hyperparameter Optimization using Supercomputing) to reduce and quantify the variance of test performance metrics of deep learning models. Methods: NACHOS integrates Nested Cross-Validation (NCV) and Automated Hyperparameter Optimization (AHPO) within a parallelized high-performance computing (HPC) framework. NACHOS was demonstrated on a chest X-ray repository and an Optical Coherence Tomography (OCT) dataset under multiple data partitioning schemes. Beyond performance estimation, DACHOS (Deployment with Automated Cross-validation and Hyperparameter Optimization using Supercomputing) is introduced to leverage AHPO and cross-validation to build the final model on the full dataset, improving expected deployment performance. Results: The findings underscore the importance of NCV in quantifying and reducing estimation variance, AHPO in optimizing hyperparameters consistently across test folds, and HPC in ensuring computational feasibility. Conclusions: By integrating these methodologies, NACHOS and DACHOS provide a scalable, reproducible, and trustworthy framework for DL model evaluation and deployment in medical imaging. To maximize public availability, the full open-source codebase is provided at https://github.com/thepanlab/NACHOS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。