用大模型增强病理与基因数据，提升癌症生存预测准确率
LLM-driven Knowledge Enhancement for Multimodal Cancer Survival Prediction
- 用大模型提炼病理报告和癌症预后知识，生成临床相关描述
- 在5个数据集上超越现有方法，显著提升生存预测性能
- 适合医学人工智能、多模态学习研究者参考
当前多模态生存预测方法主要依赖高维冗余的病理图像（WSIs）和基因组数据，难以提取判别性特征且模态对齐困难。仅用简单的随访标签也难以有效监督复杂任务。为此，我们提出KEMM——一种基于大语言模型（LLM）的知识增强多模态模型，融合专家报告与预后背景知识。1）由病理科医生提供、经大模型优化的病例级专家报告，包含简洁且临床聚焦的诊断陈述，通常暗示不同生存结局；2）大模型生成的预后背景知识（PBK），为不同癌种提供有价值的预后信息，进一步增强预测能力。为有效利用这些知识，我们设计了知识增强的跨模态注意力模块（KECM），能引导网络关注冗余模态中与生存相关的判别性特征。在五个数据集上的大量实验表明，KEMM达到当前最优性能。代码将在论文接收后公开。
原文摘要 · Abstract (English)
Current multimodal survival prediction methods typically rely on pathology images (WSIs) and genomic data, both of which are high-dimensional and redundant, making it difficult to extract discriminative features from them and align different modalities. Moreover, using a simple survival follow-up label is insufficient to supervise such a complex task. To address these challenges, we propose KEMM, an LLM-driven Knowledge-Enhanced Multimodal Model for cancer survival prediction, which integrates expert reports and prognostic background knowledge. 1) Expert reports, provided by pathologists on a case-by-case basis and refined by large language model (LLM), offer succinct and clinically focused diagnostic statements. This information may typically suggest different survival outcomes. 2) Prognostic background knowledge (PBK), generated concisely by LLM, provides valuable prognostic background knowledge on different cancer types, which also enhances survival prediction. To leverage these knowledge, we introduce the knowledge-enhanced cross-modal (KECM) attention module. KECM can effectively guide the network to focus on discriminative and survival-relevant features from highly redundant modalities. Extensive experiments on five datasets demonstrate that KEMM achieves state-of-the-art performance. The code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。