用无配对图文数据，让医疗图像分类无需标注也能提升准确率。
Can language-guided unsupervised adaptation improve medical image classification using unpaired images and texts?
- 通过文本描述与视觉编码器的跨模态适配，实现无监督学习。
- 在胸部X光等数据集上，准确率显著高于零样本基线。
- 适合标签稀缺但文本丰富的医疗场景，如放射科和皮肤科。
在医疗图像分类中，由于标注图像稀缺，监督学习面临挑战。为此，我们利用视觉-语言模型（VLMs）中的图文对齐能力，实现医疗图像分类器的无监督学习。本文提出医学无监督适配（MedUnA），将大语言模型生成的类别文本描述编码为文本嵌入，并通过跨模态适配器与类别标签匹配。该适配器连接到MedCLIP的视觉编码器，基于对比熵损失与提示调优，实现视觉嵌入的无监督对齐。相较于传统VLM，MedUnA使用无配对图像与文本学习表征，突破了原有约束。我们在三个胸部X光数据集及两个多类数据集（糖尿病视网膜病变、皮肤病变）上评估，结果表明其准确率显著优于零样本基线。代码已公开于https://github.com/rumaima/meduna。
原文摘要 · Abstract (English)
In medical image classification, supervised learning is challenging due to the scarcity of labeled medical images. To address this, we leverage the visual-textual alignment within Vision-Language Models (VLMs) to enable unsupervised learning of a medical image classifier. In this work, we propose \underline{Med}ical \underline{Un}supervised \underline{A}daptation (\texttt{MedUnA}) of VLMs, where the LLM-generated descriptions for each class are encoded into text embeddings and matched with class labels via a cross-modal adapter. This adapter attaches to a visual encoder of \texttt{MedCLIP} and aligns the visual embeddings through unsupervised learning, driven by a contrastive entropy-based loss and prompt tuning. Thereby, improving performance in scenarios where textual information is more abundant than labeled images, particularly in the healthcare domain. Unlike traditional VLMs, \texttt{MedUnA} uses \textbf{unpaired images and text} for learning representations and enhances the potential of VLMs beyond traditional constraints. We evaluate the performance on three chest X-ray datasets and two multi-class datasets (diabetic retinopathy and skin lesions), showing significant accuracy gains over the zero-shot baseline. Our code is available at https://github.com/rumaima/meduna.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。