融合病理、基因与临床数据,用注意力模型预测癌症患者生存期。
Attention-Based Multimodal Survival Prediction with Cross-Modal Bilinear Fusion

- 用注意力机制提取病理切片特征,结合基因和临床数据
- 通过低秩双线性融合,提升预测准确率且减少参数量
- 结构可解释性强,适合医疗领域多模态分析研究者
我们提出一种新型多模态深度学习框架,用于个体患者生存期预测,整合全切片组织学特征、RNA-seq表达谱及临床变量。架构结合ABMIL模块~ic{ilse2018attention}生成切片级表征,以及用于RNA和临床数据的前馈编码器。这些嵌入通过低秩双线性跨模态融合~ic{liu2018efficient}进行集成,以建模模态间的条件交互,同时控制参数增长。模型输出连续风险评分,并通过基于Kaplan–Meier估计器的非参数校准过程映射为生存时间。通过将多模态推理分解为独立的成对交互,该融合设计相比完整张量和分层融合策略更具结构可解释性和参数效率。在CHIMERA挑战赛数据集上的实验表明,其预测性能优于基于拼接的基线方法,并在隐藏评估队列上表现具有竞争力。结果表明,该框架是HR-NMIBC多模态生存预测的有前景方案。代码已公开于https://github.com/hassancpu/ChimeraChallenge2025_Task_3。
原文摘要 · Abstract (English)
We propose a novel multimodal deep learning framework for patient-level survival prediction, which integrates whole-slide histology features, RNA-seq expression profiles, and clinical variables. Our architecture combines an ABMIL module~\cite{ilse2018attention} for slide-level representation with feedforward encoders for RNA and clinical data. These embeddings are then integrated through low-rank bilinear cross-modal fusion~\cite{liu2018efficient} to model conditional interactions across modalities while controlling parameter growth. The model outputs continuous risk scores that are subsequently mapped to survival times using a nonparametric calibration procedure based on the Kaplan--Meier estimator~\cite{kaplan1958nonparametric}. By decomposing multimodal reasoning into independent pairwise interactions, the proposed fusion design promotes structural interpretability and parameter efficiency compared with full tensor and hierarchical fusion strategies. Experiments on the CHIMERA challenge dataset demonstrate improved predictive performance over concatenation-based baselines and competitive generalization on hidden evaluation cohorts. These results indicate that the proposed framework is a promising approach for multimodal survival prediction in HR-NMIBC. The implementation is publicly available at https://github.com/hassancpu/ChimeraChallenge2025_Task_3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。