arXiv:2412.00651cs.CVq-bio.GN2024-12被引 10

用基因表达数据增强病理图像学习,提升模型分子感知能力。

Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics

  • 融合空间转录组数据,引导病理图像预训练
  • 在69.7万对图像-基因数据上对齐多模态编码器
  • 适合关注分子层面病理分析的研究者

近年来,多模态预训练模型推动了计算病理学的发展。然而,现有方法主要依赖视觉-语言模型,从分子视角看存在局限性,易导致性能瓶颈。本文提出统一的分子增强病理图像表征学习框架(UMPIRE),通过基因表达谱提供互补信息,指导多模态预训练,增强病理图像表征的分子感知能力。由于配对数据稀缺,我们收集了约400万条空间转录组基因表达数据用于训练基因编码器。借助强大的预训练编码器,UMPIRE在超过69.7万张病理图像-基因表达对上实现编码器对齐。该框架在多种分子相关下游任务中表现优异,包括基因表达预测、点位分类及全切片突变状态预测。结果表明,多模态数据融合有效,为分子视角增强的计算病理学开辟新路径。代码与预训练权重已公开于https://github.com/Hanminghao/UMPIRE。

原文摘要 · Abstract (English)

Recent advancements in multimodal pre-training models have significantly advanced computational pathology. However, current approaches predominantly rely on visual-language models, which may impose limitations from a molecular perspective and lead to performance bottlenecks. Here, we introduce a Unified Molecule-enhanced Pathology Image REpresentationn Learning framework (UMPIRE). UMPIRE aims to leverage complementary information from gene expression profiles to guide the multimodal pre-training, enhancing the molecular awareness of pathology image representation learning. We demonstrate that this molecular perspective provides a robust, task-agnostic training signal for learning pathology image embeddings. Due to the scarcity of paired data, approximately 4 million entries of spatial transcriptomics gene expression were collected to train the gene encoder. By leveraging powerful pre-trained encoders, UMPIRE aligns the encoders across over 697K pathology image-gene expression pairs. The performance of UMPIRE is demonstrated across various molecular-related downstream tasks, including gene expression prediction, spot classification, and mutation state prediction in whole slide images. Our findings highlight the effectiveness of multimodal data integration and open new avenues for exploring computational pathology enhanced by molecular perspectives. The code and pre-trained weights are available at https://github.com/Hanminghao/UMPIRE.

病理图像空间转录组多模态学习分子表征

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。