用病理概念对齐实现无监督大图预训练,提升模型可解释性。
GECKO: Gigapixel Vision-Concept Contrastive Pretraining in Histopathology
- 基于病理概念相似性构建视觉-概念先验,无需额外标注数据
- 双分支对比学习使全切片嵌入与概念嵌入对齐,性能优于现有方法
- 适用于缺乏多模态数据的病理图像研究,适合临床可解释性需求
多实例学习(MIL)聚合器可无监督地从切片块中生成全切片图像(WSI)级嵌入。尽管近期利用辅助模态的多模态MIL预训练方法在性能上优于单模态方法,但获取这些额外模态需大量临床标注,增加成本并限制了在缺乏配对模态的现有数据集中的可扩展性。为此,我们提出巨像素视觉-概念知识对比预训练(GECKO),将WSI与从可用WSI中推导出的概念先验对齐。首先,通过计算每个切片块与预定义病理概念的文本描述之间的相似性,构建内在可解释的概念先验。GECKO采用双分支MIL网络:一分支将切片块嵌入聚合为WSI级深度嵌入,另一分支将概念先验聚合为对应的WSI级概念嵌入。两者通过对比目标对齐,从而完成整个双分支MIL模型的预训练。此外,当有转录组等辅助模态可用时,GECKO可无缝集成。在五个不同任务中,GECKO持续优于先前的单模态和多模态预训练方法,并实现了临床有意义的可解释性,弥合了计算模型与病理学专家之间的差距。代码已开源于https://github.com/bmi-imaginelab/GECKO。
原文摘要 · Abstract (English)
Pretraining a Multiple Instance Learning (MIL) aggregator enables the derivation of Whole Slide Image (WSI)-level embeddings from patch-level representations without supervision. While recent multimodal MIL pretraining approaches leveraging auxiliary modalities have demonstrated performance gains over unimodal WSI pretraining, the acquisition of these additional modalities necessitates extensive clinical profiling. This requirement increases costs and limits scalability in existing WSI datasets lacking such paired modalities. To address this, we propose Gigapixel Vision-Concept Knowledge Contrastive pretraining (GECKO), which aligns WSIs with a Concept Prior derived from the available WSIs. First, we derive an inherently interpretable concept prior by computing the similarity between each WSI patch and textual descriptions of predefined pathology concepts. GECKO then employs a dual-branch MIL network: one branch aggregates patch embeddings into a WSI-level deep embedding, while the other aggregates the concept prior into a corresponding WSI-level concept embedding. Both aggregated embeddings are aligned using a contrastive objective, thereby pretraining the entire dual-branch MIL model. Moreover, when auxiliary modalities such as transcriptomics data are available, GECKO seamlessly integrates them. Across five diverse tasks, GECKO consistently outperforms prior unimodal and multimodal pretraining approaches while also delivering clinically meaningful interpretability that bridges the gap between computational models and pathology expertise. Code is made available at https://github.com/bmi-imaginelab/GECKO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。