通过语义感知对比学习,提升3D医学影像中的假负例识别能力。
Multimodal Semantic-Aware Contrastive Learning For False Negative Mitigation in 3D Medical Imaging

- 基于报告语义相似性动态调整负样本,减少假负例干扰。
- 在儿科脑肿瘤分子分类任务中AUC提升至少22.6%。
- 适合需要高精度多模态医疗表征的临床研究与系统开发。
多模态对比学习在对齐不同数据模态表示、提升下游任务性能方面表现优异,尤其在医疗领域。其核心是缩小匹配样本(正例)间的距离,扩大不匹配样本(负例)间的距离。传统框架假设批次内所有非配对样本均为负例,但在医学场景中,样本可能具有高层语义相似性,导致假负例出现,损害表征质量。本文提出多模态语义感知对比学习(MseaCL),在儿童3D脑部MRI扫描与放射科报告的队列上训练,利用报告间的语义相似性作为学习过程中的引导信号,缓解语义相似但未配对样本带来的负面影响。实验表明,将该框架作为预训练阶段可显著提升下游任务性能,例如在儿童脑肿瘤分子分类任务中,受试者工作特征曲线下面积(AUC)至少提升22.6%,展现出在临床应用中构建更鲁棒、语义对齐的多模态表征的潜力。
原文摘要 · Abstract (English)
Multimodal Contrastive Learning (CL) has shown significant performance in aligning representations across various data modalities and improving downstream tasks, especially in healthcare. It works by minimizing the distance between matched (positive) data modalities, while maximizing the distance between mismatched (negative) samples. Traditional CL frameworks typically assume instance-based correspondence within data batches, treating all non-paired samples as negatives. However, this assumption often fails in medical settings, where samples may share high-level semantic attributes, leading to false negatives that degrade representation quality. In this paper, we propose Multimodal Semantic-Aware Contrastive Learning (MseaCL), a CL framework trained on a pediatric cohort of 3D brain magnetic resonance imaging (MRI) scans and radiology reports. The goal of this framework is to mitigate the impact of semantically similar false negative samples by incorporating semantic similarity between radiology reports, as a guiding signal during the learning process. Our results indicate that applying this framework as a pretraining stage can achieve notable improvements in downstream tasks, e.g., at least a 22.6\% increase in the area under the receiver operating characteristic curve (AUC) of pediatric brain tumor molecular classification, demonstrating its potential for more robust and semantically aligned multimodal representations in clinical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。