用跨模态掩码学习融合影像与临床数据，提升免疫治疗肺癌患者生存预测准确率。
Cross-Modality Masked Learning for Survival Prediction in ICI Treated NSCLC Patients
- 设计双分支模型：3D CT用切片深度变换器，临床数据用图注意力网络
- 通过掩码重建机制，使不同模态间特征互补，提升融合效果
- 在真实临床数据上表现最优，适合精准医疗和肿瘤预后研究者
准确预测接受免疫检查点抑制剂（ICI）治疗的非小细胞肺癌（NSCLC）患者的预后，对个性化治疗决策、提高治疗效果和生活质量至关重要。然而，缺乏大规模相关数据集及有效的多模态特征融合方法成为主要挑战。为此，我们构建了一个大规模数据集，包含接受ICI治疗的NSCLC患者的3D CT影像与临床记录，并附有无进展生存期（PFS）和总生存期（OS）数据。提出一种跨模态掩码学习框架用于医学特征融合，包含两个独立分支：针对CT影像的切片-深度变换器（Slice-Depth Transformer）提取三维特征，针对表格型临床数据的图结构变换器（graph-based Transformer）学习变量间的节点特征与关系。融合过程采用掩码模态学习策略，利用完整模态重建缺失部分，增强模态特异性特征整合，促进跨模态关联与交互。实验表明，该方法在多模态融合方面显著优于现有方法，为该领域设定新基准。
原文摘要 · Abstract (English)
Accurate prognosis of non-small cell lung cancer (NSCLC) patients undergoing immunotherapy is essential for personalized treatment planning, enabling informed patient decisions, and improving both treatment outcomes and quality of life. However, the lack of large, relevant datasets and effective multi-modal feature fusion strategies pose significant challenges in this domain. To address these challenges, we present a large-scale dataset and introduce a novel framework for multi-modal feature fusion aimed at enhancing the accuracy of survival prediction. The dataset comprises 3D CT images and corresponding clinical records from NSCLC patients treated with immune checkpoint inhibitors (ICI), along with progression-free survival (PFS) and overall survival (OS) data. We further propose a cross-modality masked learning approach for medical feature fusion, consisting of two distinct branches, each tailored to its respective modality: a Slice-Depth Transformer for extracting 3D features from CT images and a graph-based Transformer for learning node features and relationships among clinical variables in tabular data. The fusion process is guided by a masked modality learning strategy, wherein the model utilizes the intact modality to reconstruct missing components. This mechanism improves the integration of modality-specific features, fostering more effective inter-modality relationships and feature interactions. Our approach demonstrates superior performance in multi-modal integration for NSCLC survival prediction, surpassing existing methods and setting a new benchmark for prognostic models in this context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。