用多模态自监督模型融合脑影像与临床数据,提升中风风险预测准确率。
Advancing Stroke Risk Prediction Using a Multi-modal Foundation Model
- 通过对比学习对齐图像与表格数据的表征,实现跨模态信息融合。
- 在英国生物银行数据上,平衡准确率比最优监督模型高7.6%。
- 可解释性分析揭示了与脑衰老和中风相关的关键脑区激活。
中风风险预测是一项复杂挑战,可通过整合多种临床可用数据模态来提升。本研究提出一种自监督多模态框架,结合3D脑影像、临床数据与图像衍生特征,以改善发病前的中风风险预测。该框架利用大规模未标注临床数据集,捕捉图像与表格数据间的互补与协同信息。方法基于对比学习,融合对比语言-图像预训练与图像-表格匹配模块,将多模态数据表示对齐至共享潜在空间。模型在包含结构化脑部MRI与临床数据的英国生物银行(UK Biobank)上训练。我们在多种冻结与可训练设置下,针对表格、图像及图像-表格组合,对比了当前先进单模态与多模态方法。所提模型在ROC-AUC上较自监督表格(图像)方法分别提升2.6%(2.6%),在平衡准确率上提升3.3%(5.6%)。相较于最优的多模态监督模型,平衡准确率提升7.6%。通过可解释工具,该方法展示了更优的表格与图像数据融合能力,生成更丰富且对齐的嵌入表示。梯度加权类激活映射热图进一步揭示了文献中常见于脑衰老、中风风险与临床结果的脑区激活。这一稳健的自监督多模态框架超越现有最优方法,在中风风险预测中表现优异,并为未来融合多样化数据模态推进临床预测建模提供坚实基础。
原文摘要 · Abstract (English)
Predicting stroke risk is a complex challenge that can be enhanced by integrating diverse clinically available data modalities. This study introduces a self-supervised multimodal framework that combines 3D brain imaging, clinical data, and image-derived features to improve stroke risk prediction prior to onset. By leveraging large unannotated clinical datasets, the framework captures complementary and synergistic information across image and tabular data modalities. Our approach is based on a contrastive learning framework that couples contrastive language-image pretraining with an image-tabular matching module, to better align multimodal data representations in a shared latent space. The model is trained on the UK Biobank, which includes structural brain MRI and clinical data. We benchmark its performance against state-of-the-art unimodal and multimodal methods using tabular, image, and image-tabular combinations under diverse frozen and trainable model settings. The proposed model outperformed self-supervised tabular (image) methods by 2.6% (2.6%) in ROC-AUC and by 3.3% (5.6%) in balanced accuracy. Additionally, it showed a 7.6% increase in balanced accuracy compared to the best multimodal supervised model. Through interpretable tools, our approach demonstrated better integration of tabular and image data, providing richer and more aligned embeddings. Gradient-weighted Class Activation Mapping heatmaps further revealed activated brain regions commonly associated in the literature with brain aging, stroke risk, and clinical outcomes. This robust self-supervised multimodal framework surpasses state-of-the-art methods for stroke risk prediction and offers a strong foundation for future studies integrating diverse data modalities to advance clinical predictive modelling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。