arXiv:2409.07960cs.CVcs.LG2024-09ECCV被引 11

用大模型提升医疗影像分割的跨域泛化能力,效果显著。

Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?

  • 用多种大模型+高效微调技术,在多数据集上测试其跨域表现。
  • 新设计的HQHSAM解码头使分割精度提升,尤其在不同设备间表现更稳定。
  • 发现不同大模型对微调方法敏感度不同,为临床应用提供参考。

神经网络在训练与测试数据分布一致时表现优异,但在医学图像分割中因扫描仪型号和协议差异导致的领域偏移(协变量转移)下性能显著下降。近年来,基于大规模数据训练的通用模型(FMs)在自然图像任务中展现出强大泛化能力,但其在医学图像分割中的有效性尚不明确。本文研究了DinoV2、SAM、MedSAM和MAE等通用模型,在采用Ladder、Rein(+LoRA)等参数高效微调(PEFT)技术及不同解码头结构时的领域泛化能力。提出一种新型解码头HQHSAM,融合HSAM与HQSAM的优势。在涵盖多种解剖结构与成像模态的多个数据集上进行大量实验,结果表明,使用通用模型尤其是搭配HQHSAM解码头可显著提升医学图像分割的领域泛化性能。此外,不同模型对微调策略的响应存在差异。这些发现证明通用模型有望增强神经网络在多样临床环境下的泛化能力,为后续研究奠定基础。代码与模型已开源。

原文摘要 · Abstract (English)

Neural networks achieve state-of-the-art performance in many supervised learning tasks when the training data distribution matches the test data distribution. However, their performance drops significantly under domain (covariate) shift, a prevalent issue in medical image segmentation due to varying acquisition settings across different scanner models and protocols. Recently, foundational models (FMs) trained on large datasets have gained attention for their ability to be adapted for downstream tasks and achieve state-of-the-art performance with excellent generalization capabilities on natural images. However, their effectiveness in medical image segmentation remains underexplored. In this paper, we investigate the domain generalization performance of various FMs, including DinoV2, SAM, MedSAM, and MAE, when fine-tuned using various parameter-efficient fine-tuning (PEFT) techniques such as Ladder and Rein (+LoRA) and decoder heads. We introduce a novel decode head architecture, HQHSAM, which simply integrates elements from two state-of-the-art decoder heads, HSAM and HQSAM, to enhance segmentation performance. Our extensive experiments on multiple datasets, encompassing various anatomies and modalities, reveal that FMs, particularly with the HQHSAM decode head, improve domain generalization for medical image segmentation. Moreover, we found that the effectiveness of PEFT techniques varies across different FMs. These findings underscore the potential of FMs to enhance the domain generalization performance of neural networks in medical image segmentation across diverse clinical settings, providing a solid foundation for future research. Code and models are available for research purposes at \url{https://github.com/kerem-cekmeceli/Foundation-Models-for-Medical-Imagery}.

医学影像大模型泛化能力分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。