arXiv:2409.09430cs.CV2024-09被引 32

对比了多种模型在医学图像检索中的表现,发现大模型效果更优。

Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval

  • 用预训练CNN和基础模型提取医学图像特征
  • 大模型在2D图像上显著优于CNN,CONCH在3D上最佳
  • 小尺寸图像也能达到良好检索效果,适合资源受限场景

医学图像检索旨在从数据库中为查询图像找到相似图像,用于辅助诊断。传统方法依赖临床元数据,而基于内容的医学图像检索(CBMIR)则依赖自动提取的图像特征。本文评估了多个知名预训练CNN和基础模型在八类二维(2D)与三维(3D)医学图像上的CBMIR性能,并研究了图像尺寸的影响。结果表明:在2D数据集上,基础模型整体表现远超CNN,通用自监督病理模型UNI在所有数据集和尺寸下表现最佳;在3D数据集上,CNN与基础模型表现接近,基于组织学图文对比学习的模型CONCH表现最优。此外,尽管较大图像尺寸能略微提升性能,但小尺寸图像仍可实现竞争力的检索效果。代码已公开于https://github.com/masih4/MedImageRetrieval。

原文摘要 · Abstract (English)

Medical image retrieval refers to the task of finding similar images for given query images in a database, with applications such as diagnosis support. While traditional medical image retrieval relied on clinical metadata, content-based medical image retrieval (CBMIR) depends on image features, which can be extracted automatically or semi-automatically. Many approaches have been proposed for CBMIR, and among them, using pre-trained convolutional neural networks (CNNs) is a widely utilized approach. However, considering the recent advances in the development of foundation models for various computer vision tasks, their application for CBMIR can also be investigated. In this study, we used several pre-trained feature extractors from well-known pre-trained CNNs and pre-trained foundation models and investigated the CBMIR performance on eight types of two-dimensional (2D) and three-dimensional (3D) medical images. Furthermore, we investigated the effect of image size on the CBMIR performance. Our results show that, overall, for the 2D datasets, foundation models deliver superior performance by a large margin compared to CNNs, with the general-purpose self-supervised model for computational pathology (UNI) providing the best overall performance across all datasets and image sizes. For 3D datasets, CNNs and foundation models deliver more competitive performance, with contrastive learning from captions for histopathology model (CONCH) achieving the best overall performance. Moreover, our findings confirm that while using larger image sizes (especially for 2D datasets) yields slightly better performance, competitive CBMIR performance can still be achieved even with smaller image sizes. Our codes to reproduce the results are available at: https://github.com/masih4/MedImageRetrieval.

医学图像特征提取大模型检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。