arXiv:2601.05399cs.CVcs.AI2026-01

用多任务学习优化医学影像检索,提升肺部X光片与报告的匹配精度。

Multi-task Cross-modal Learning for Chest X-ray Image Retrieval

  • 基于BiomedCLIP设计多任务损失,融合分类、对比和跨模态对齐训练。
  • 在图像到文本和文本到图像检索中均优于预训练模型,性能更均衡。
  • 适合医疗AI研发者,尤其关注医学影像与报告精准匹配场景。

CLIP和BiomedCLIP是视觉-语言基础模型,具备强大的跨模态嵌入能力;然而它们未针对细粒度医学检索任务(如使用胸部X光片查询临床相关放射科报告)进行优化。为此,我们提出一种多任务学习框架,对BiomedCLIP进行微调,并评估其在胸部X光片图文检索上的改进效果。以BiomedCLIP为骨干网络,引入轻量级MLP投影头,通过包含三部分的复合损失函数进行训练:(1) 二值交叉熵损失,用于区分正常与异常的胸片;(2) 监督对比损失,强化同类样本的一致性;(3) CLIP损失,保持跨模态对齐。实验表明,微调后模型在图像到文本和文本到图像检索任务中均表现更平衡、更具临床意义,优于预训练的BiomedCLIP及通用型CLIP模型。此外,t-SNE可视化显示正常与异常病例的语义聚类更清晰,证明模型诊断敏感性增强。这些结果凸显了领域自适应多任务学习在生物医学跨模态检索中的价值。

原文摘要 · Abstract (English)

CLIP and BiomedCLIP are examples of vision-language foundation models and offer strong cross-modal embeddings; however, they are not optimized for fine-grained medical retrieval tasks, such as retrieving clinically relevant radiology reports using chest X-ray (CXR) image queries. To address this shortcoming, we propose a multi-task learning framework to fine-tune BiomedCLIP and evaluate improvements to CXR image-text retrieval. Using BiomedCLIP as the backbone, we incorporate a lightweight MLP projector head trained with a multi-task composite loss function that includes: (1) a binary cross-entropy loss to distinguish normal from abnormal CXR studies, (2) a supervised contrastive loss to reinforce intra-class consistency, and (3) a CLIP loss to maintain cross-modal alignment. Experimental results demonstrate that the fine-tuned model achieves more balanced and clinically meaningful performance across both image-to-text and text-to-image retrieval tasks compared to the pretrained BiomedCLIP and general-purpose CLIP models. Furthermore, t-SNE visualizations reveal clearer semantic clustering of normal and abnormal cases, demonstrating the model's enhanced diagnostic sensitivity. These findings highlight the value of domain-adaptive, multi-task learning for advancing cross-modal retrieval in biomedical applications.

医学影像跨模态多任务学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。