首次解析乳腺X光模型如何学习临床概念,揭示训练数据对模型理解力的影响。
Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language Models
- 用专用于乳腺X光的视觉语言模型定位神经元对应的人类可读概念
- 发现乳腺影像训练数据能提升模型与放射科医生工作流程的契合度
- 适合关注AI医疗可解释性、乳腺癌筛查模型优化的研究者
深度学习模型的理解能力对人工智能在临床场景的安全部署至关重要。以往研究多关注像素级可解释性,较少关注模型所学习的文本概念,而这些概念更贴近临床决策逻辑。本文提出Mammo-CLIP Dissect,首个针对乳腺影像视觉模型的概念基础可解释性框架。利用专为乳腺影像设计的视觉语言模型(Mammo-CLIP)作为“解剖工具”,该方法在特定层标注神经元对应的人类可读文本概念,并量化其与领域知识的对齐程度。通过该框架,我们探讨了三个关键问题:(1) 在通用图像数据集与乳腺影像数据集上训练的模型在概念学习上的差异;(2) 针对下游任务微调如何影响概念特异性;(3) 哪些乳腺影像相关概念仍被忽视。结果表明,使用乳腺影像数据训练的模型能捕捉更多临床相关概念,且与放射科医生的工作流程更一致。任务特定微调虽增强了某些概念(如良性钙化)的表达,但削弱了其他概念(如密度特征)的覆盖,反映出专业化与泛化之间的权衡。本研究揭示了卷积神经网络如何捕获乳腺影像特有知识,通过对比不同训练数据与微调策略,阐明了领域特异性训练和任务适配对概念学习的影响。代码与概念集已开源:https://github.com/Suaiba/Mammo-CLIP-Dissect。
原文摘要 · Abstract (English)
Understanding what deep learning (DL) models learn is essential for the safe deployment of artificial intelligence (AI) in clinical settings. While previous work has focused on pixel-based explainability methods, less attention has been paid to the textual concepts learned by these models, which may better reflect the reasoning used by clinicians. We introduce Mammo-CLIP Dissect, the first concept-based explainability framework for systematically dissecting DL vision models trained for mammography. Leveraging a mammography-specific vision-language model (Mammo-CLIP) as a "dissector," our approach labels neurons at specified layers with human-interpretable textual concepts and quantifies their alignment to domain knowledge. Using Mammo-CLIP Dissect, we investigate three key questions: (1) how concept learning differs between DL vision models trained on general image datasets versus mammography-specific datasets; (2) how fine-tuning for downstream mammography tasks affects concept specialisation; and (3) which mammography-relevant concepts remain underrepresented. We show that models trained on mammography data capture more clinically relevant concepts and align more closely with radiologists' workflows than models not trained on mammography data. Fine-tuning for task-specific classification enhances the capture of certain concept categories (e.g., benign calcifications) but can reduce coverage of others (e.g., density-related features), indicating a trade-off between specialisation and generalisation. Our findings show that Mammo-CLIP Dissect provides insights into how convolutional neural networks (CNNs) capture mammography-specific knowledge. By comparing models across training data and fine-tuning regimes, we reveal how domain-specific training and task-specific adaptation shape concept learning. Code and concept set are available: https://github.com/Suaiba/Mammo-CLIP-Dissect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。