通过跨模态聚类引导负样本采样，提升医学影像与报告联合自监督学习效果
Cross-Modal Clustering-Guided Negative Sampling for Self-Supervised Joint Learning from Medical Images and Reports
- 利用跨模态注意力将聚类扩展至多模态，增强负样本多样性
- 在五大数据集上分类、检测、分割任务均优于现有方法
- 适合需要细粒度医学图像理解的研究者和临床辅助诊断系统开发者
近年来，通过多模态自监督学习直接从配对的医学影像与报告中学习视觉表征，已成为数字诊断的新颖高效方法。然而，现有模型存在诸多严重局限：1）忽略负样本选择，导致难负样本稀缺且包含错误负样本；2）侧重全局特征提取，忽视对医学图像识别至关重要的细粒度局部细节；3）对比学习主要关注高层特征，忽略对精准医疗分析至关重要的低层细节。针对这些问题，本文提出一种跨模态聚类引导负样本采样（CM-CGNS）方法，包含两方面创新：首先，通过跨模态注意力将单模态局部文本特征的k-means聚类拓展至多模态域，增加负样本数量并提升模型表示能力；其次，引入跨模态掩码图像重建（CM-MIR）模块，利用跨模态注意力获取的局部图文特征重建被掩码的图像区域，显著强化跨模态信息交互能力，并保留下游任务所需的低层图像特征。通过有效解决上述问题，所提方法能学习到适用于多种识别任务的有效且鲁棒的医学视觉表征。在五个下游数据集上的分类、检测与分割任务的大量实验表明，该方法在多个指标上优于当前最优方法，验证了其优越性能。
原文摘要 · Abstract (English)
Learning medical visual representations directly from paired images and reports through multimodal self-supervised learning has emerged as a novel and efficient approach to digital diagnosis in recent years. However, existing models suffer from several severe limitations. 1) neglecting the selection of negative samples, resulting in the scarcity of hard negatives and the inclusion of false negatives; 2) focusing on global feature extraction, but overlooking the fine-grained local details that are crucial for medical image recognition tasks; and 3) contrastive learning primarily targets high-level features but ignoring low-level details which are essential for accurate medical analysis. Motivated by these critical issues, this paper presents a Cross-Modal Cluster-Guided Negative Sampling (CM-CGNS) method with two-fold ideas. First, it extends the k-means clustering used for local text features in the single-modal domain to the multimodal domain through cross-modal attention. This improvement increases the number of negative samples and boosts the model representation capability. Second, it introduces a Cross-Modal Masked Image Reconstruction (CM-MIR) module that leverages local text-to-image features obtained via cross-modal attention to reconstruct masked local image regions. This module significantly strengthens the model's cross-modal information interaction capabilities and retains low-level image features essential for downstream tasks. By well handling the aforementioned limitations, the proposed CM-CGNS can learn effective and robust medical visual representations suitable for various recognition tasks. Extensive experimental results on classification, detection, and segmentation tasks across five downstream datasets show that our method outperforms state-of-the-art approaches on multiple metrics, verifying its superior performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。