arXiv:2411.17418cs.CV2024-11

通过双阶段融合病理图像与甲基化数据,提升脑肿瘤精准分类效果。

Multimodal Outer Arithmetic Block Dual Fusion of Whole Slide Images and Omics Data for Precision Oncology

  • 早期将基因数据投影到图像局部区域,晚期用外积算子重构多模态特征
  • 在TCGA-BLCA上生存预测优于现有方法,在20种亚型分类中表现优异
  • 适合需要可解释性与高精度的神经肿瘤病理诊断场景

DNA甲基化数据与全切片图像(WSI)的融合对中枢神经系统(CNS)肿瘤分类具有重要价值。现有方法通常仅在早期或晚期进行多模态融合,未探索将基因信息通过双重融合重新引入的可能性。本文提出在早期和晚期均融合基因嵌入,以捕捉从局部(切片级)到全局(整片级)的互补信息。早期阶段,将基因嵌入投影至WSI切片的潜在空间,生成融合分子与形态信息的嵌入表示,有效整合基因特征于空间结构中;随后通过多重实例学习门控注意力机制聚焦诊断性切片。晚期阶段,利用多模态外积算子(MOAB)将基因数据与滑片级基因-图像嵌入融合,深度交织双模态特征,捕获其相关性与互补性。我们在20种细粒度亚型上验证了该方法,实验结果表明在TCGA-BLCA上生存预测性能优于现有方法,在TCGA-BRCA上达到竞争力水平。该双融合策略显著提升了分类准确率与可解释性,展现出临床诊断应用潜力。

原文摘要 · Abstract (English)

The integration of DNA methylation data with a Whole Slide Image (WSI) offers significant potential for enhancing the diagnostic precision of central nervous system (CNS) tumor classification in neuropathology. While existing approaches typically integrate encoded omic data with histology at either an early or late fusion stage, the potential of reintroducing omic data through dual fusion remains unexplored. In this paper, we propose the use of omic embeddings during early and late fusion to capture complementary information from local (patch-level) to global (slide-level) interactions, boosting performance through multimodal integration. In the early fusion stage, omic embeddings are projected onto WSI patches in latent-space, which generates embeddings that encapsulate per-patch molecular and morphological insights. This effectively incorporates omic information into the spatial representation of the WSI. These embeddings are then refined with a Multiple Instance Learning gated attention mechanism which attends to diagnostic patches. In the late fusion stage, we reintroduce the omic data by fusing it with slide-level omic-WSI embeddings using a Multimodal Outer Arithmetic Block (MOAB), which richly intermingles features from both modalities, capturing their correlations and complementarity. We demonstrate accurate CNS tumor subtyping across 20 fine-grained subtypes and validate our approach on benchmark datasets, achieving improved survival prediction on TCGA-BLCA and competitive performance on TCGA-BRCA compared to state-of-the-art methods. This dual fusion strategy enhances interpretability and classification performance, highlighting its potential for clinical diagnostics.

多模态融合病理图像精准肿瘤学基因组数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。