arXiv:2604.03635cs.CVcs.AI2026-04被引 1

一个统一模型能生成多种病理数据,提升诊断效率与准确性。

A Generative Foundation Model for Multimodal Histopathology

  • 用扩散Transformer将病理图像、分子数据和临床文本融合到同一空间。
  • 生成的组织结构更真实,比专用模型FID降低50%,少样本分类准确率提升47%。
  • 适合病理研究、医学影像生成和数据补全,尤其在数据稀缺时有用。

复杂疾病诊断需整合组织学、分子和临床数据,但因组织稀缺、检测成本高和流程限制,多模态数据常不完整。现有方法依赖特定任务模型,仅处理单一数据对,泛化能力差。本文提出MuPD(多模态病理扩散模型),通过解耦交叉注意力的扩散Transformer,将苏木精-伊红染色组织图像、分子RNA谱和临床文本嵌入共享潜在空间。模型在1亿张组织图像块、160万图文对及1080万RNA-图像对(覆盖34种人体器官)上预训练,支持多种跨模态生成任务,几乎无需微调。在文本条件生成和图像到图像生成中,其生成的组织结构更逼真,相比领域专用模型FID降低50%,并通过合成数据增强使少样本分类准确率最高提升47%。在RNA条件生成中,相较最优方法FID降低23%,且在五种癌症类型中保持细胞类型分布。作为虚拟染色器,可将H&E图像转为免疫组化和多重荧光图像,平均标记相关性提升37%。结果表明,一个跨异构病理模态预训练的统一生成模型,显著优于专用模型,为多模态病理提供可扩展的计算框架。

原文摘要 · Abstract (English)

Accurate diagnosis and treatment of complex diseases require integrating histological, molecular, and clinical data, yet in practice these modalities are often incomplete owing to tissue scarcity, assay cost, and workflow constraints. Existing computational approaches attempt to impute missing modalities from available data but rely on task-specific models trained on narrow, single source-target pairs, limiting their generalizability. Here we introduce MuPD (Multimodal Pathology Diffusion), a generative foundation model that embeds hematoxylin and eosin (H&E)-stained histology, molecular RNA profiles, and clinical text into a shared latent space through a diffusion transformer with decoupled cross-modal attention. Pretrained on 100 million histology image patches, 1.6 million text-histology pairs, and 10.8 million RNA-histology pairs spanning 34 human organs, MuPD supports diverse cross-modal synthesis tasks with minimal or no task-specific fine-tuning. For text-conditioned and image-to-image generation, MuPD synthesizes histologically faithful tissue architectures, reducing Fréchet inception distance (FID) scores by 50% relative to domain-specific models and improving few-shot classification accuracy by up to 47% through synthetic data augmentation. For RNA-conditioned histology generation, MuPD reduces FID by 23% compared with the next-best method while preserving cell-type distributions across five cancer types. As a virtual stainer, MuPD translates H&E images to immunohistochemistry and multiplex immunofluorescence, improving average marker correlation by 37% over existing approaches. These results demonstrate that a single, unified generative model pretrained across heterogeneous pathology modalities can substantially outperform specialized alternatives, providing a scalable computational framework for multimodal histopathology.

病理生成多模态扩散模型医学AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。