用乳腺影像图谱文本提升模型对BI-RADS分级的判断能力
MAM-CLIP: Vision-Language Pretraining on Mammography Atlases for BI-RADS Classification
- 用图谱文本+对比学习训练视觉语言模型
- 少样本下F1提升最高达14%
- 图文对比训练比纯标签数据更有效
深度学习在从乳腺钼靶图像预测BI-RADS评分方面已展现良好效果，但图像解读存在差异，甚至放射科医生间也常不一致。鉴于乳腺影像的复杂性，仅依赖图像标签训练分类模型往往效果有限。为此，我们从两个乳腺影像图谱中收集了2313张钼靶图像及其对应描述文本。提出一种多模态模型，采用预训练PubMedBERT作为语言模块，通过图像-文本对进行对比学习训练，使视觉编码器吸收文本中的丰富信息，从而提升对乳腺影像表现的理解能力。随后在两个数据集上微调视觉编码器进行BI-RADS预测，相比无此预训练的模型表现更优，尤其在标注样本稀缺时。3类平均F1得分提升1%至14%：4万样本下提升1%，1千样本下提升14%。实验还发现，仅2千张图-文对提供的信息量超过2千个带标签样本，当训练样本超1万时平均提升1.1%。本工作构建了乳腺影像的视觉语言模型，并揭示图谱文本的价值。此外，我们公开发布TEKNOFEST数据集的预处理图像。训练代码、预训练权重、数据提取脚本及数据集均已开源：https://github.com/igulluk/MAM-CLIP
原文摘要 · Abstract (English)
Deep learning methods have demonstrated promising results in predicting BI-RADS scores from mammography images. However, the interpretation of these images can vary, leading to discrepancies even among radiologists. Given the inherent complexity of mammograms, training classification models solely on image labels often yields limited performance. To address this challenge, we curated 2313 mammogram images and their corresponding captions from two mammography atlases. Our proposed approach employs a multi-modal model that uses a pretrained PubMedBERT as the language component. By training this model on image-text pairs with contrastive learning, we enable the vision encoder to absorb the rich information contained in the captions, thereby improving its understanding of mammography findings. We then fine-tune the vision encoder on two datasets for BI-RADS prediction, achieving superior performance compared with models trained without this pretraining, particularly when labeled samples are scarce. The improvement in the 3-class average F1 score ranges from +1% to +14%: a +1% increase with 40K training samples, and a +14% increase with 1K samples. Furthermore, our experiments reveal that 2K image-text pairs from mammography atlases can be more informative than 2K labeled samples for label prediction, with an average margin of +1.1% when more than 10K training samples are available. Overall, our work provides a vision-language model for mammography and highlights the value of textual information from mammography atlases. In addition, we publicly release preprocessed mammography images of the TEKNOFEST dataset. The training code, pre-trained model weights, data extraction scripts, and the released dataset are publicly available at: https://github.com/igulluk/MAM-CLIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。