用分割模型引导注意力，提升医学图像少样本分类精度
MedFocusCLIP : Improving few shot classification in medical datasets using pixel wise attention
- 用SAM2生成像素级提示，指导CLIP关注医学图像关键区域
- 少样本下准确率最高提升17%，在乳腺癌数据集上达58%
- 结果可解释性强，定位信息能说明分类依据
随着基础模型的普及，参数高效微调已成为利用预训练模型完成下游任务的标准方法。受大语言模型进展启发，视觉提示调优等技术通过学习额外提示来高效微调预训练视觉模型。然而我们发现，这类提示在细粒度医学图像分类任务中表现不足，因类别间差异大、类内差异小。为此，本文提出利用Segment Anything Model 2（SAM2）的先进分割能力作为视觉提示，引导CLIP视觉编码器关注图像中的相关区域，帮助模型聚焦于高区分性区域，避免被视觉相似的背景干扰。该方法在多种医学数据集（包括X光、CT和MRI）上验证，少样本分类准确率分别达到（71%、81%、86%、58%），优于仅微调CLIP的基线（66%、70%、68%、29%）。此外，通过分割得到的定位信息可提供分类结果的可解释性说明。
原文摘要 · Abstract (English)
With the popularity of foundational models, parameter efficient fine tuning has become the defacto approach to leverage pretrained models to perform downstream tasks. Taking inspiration from recent advances in large language models, Visual Prompt Tuning, and similar techniques, learn an additional prompt to efficiently finetune a pretrained vision foundational model. However, we observe that such prompting is insufficient for fine-grained visual classification tasks such as medical image classification, where there is large inter-class variance, and small intra-class variance. Hence, in this paper we propose to leverage advanced segmentation capabilities of Segment Anything Model 2 (SAM2) as a visual prompting cue to help visual encoder in the CLIP (Contrastive Language-Image Pretraining) by guiding the attention in CLIP visual encoder to relevant regions in the image. This helps the model to focus on highly discriminative regions, without getting distracted from visually similar background features, an essential requirement in a fewshot, finegrained classification setting. We evaluate our method on diverse medical datasets including X-rays, CT scans, and MRI images, and report an accuracy of (71%, 81%, 86%, 58%) from the proposed approach on (COVID, lung-disease, brain-tumor, breast-cancer) datasets against (66%, 70%, 68%, 29%) from a pretrained CLIP model after fewshot training. The proposed approach also allows to obtain interpretable explanation for the classification performance through the localization obtained using segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。