融合多模态数据预测轻度认知障碍转化,解决缺失脑部影像问题。
ITCFN: Incomplete Triple-Modal Co-Attention Fusion Network for Mild Cognitive Impairment Conversion Prediction
- 用MRI生成缺失PET数据,结合三模态注意力融合提升预测能力。
- 在ADNI1/ADNI2数据集上准确率超现有模型10%以上。
- 适合做阿尔茨海默病早期预警的研究者和临床医生参考。
阿尔茨海默病(AD)是老年人群中常见的神经退行性疾病。早期预测其前驱阶段——轻度认知障碍(MCI)的转化,有助于降低发展为AD的风险。多模态信息融合可显著提升预测准确性,但缺失数据和模态异质性使多模态学习复杂化,增加更多模态反而加剧问题。现有融合方法难以适应医疗数据的复杂性,无法有效挖掘模态间关系。为此,我们提出一种针对缺失正电子发射断层扫描(PET)数据的不完整三模态共注意力融合网络(ITCFN),用于预测MCI转化。通过缺失模态生成模块,从磁共振成像(MRI)中合成缺失的PET数据,并使用定制编码器提取特征;设计通道聚合模块与三模态共注意力融合模块,减少特征冗余,实现高效融合;同时构建损失函数处理缺失模态并对齐跨模态特征。上述组件协同利用多模态数据提升网络性能。在ADNI1和ADNI2数据集上的实验表明,该方法显著优于现有单模态及其他多模态模型。代码已开源:https://github.com/justinhxy/ITFC。
原文摘要 · Abstract (English)
Alzheimer's disease (AD) is a common neurodegenerative disease among the elderly. Early prediction and timely intervention of its prodromal stage, mild cognitive impairment (MCI), can decrease the risk of advancing to AD. Combining information from various modalities can significantly improve predictive accuracy. However, challenges such as missing data and heterogeneity across modalities complicate multimodal learning methods as adding more modalities can worsen these issues. Current multimodal fusion techniques often fail to adapt to the complexity of medical data, hindering the ability to identify relationships between modalities. To address these challenges, we propose an innovative multimodal approach for predicting MCI conversion, focusing specifically on the issues of missing positron emission tomography (PET) data and integrating diverse medical information. The proposed incomplete triple-modal MCI conversion prediction network is tailored for this purpose. Through the missing modal generation module, we synthesize the missing PET data from the magnetic resonance imaging and extract features using specifically designed encoders. We also develop a channel aggregation module and a triple-modal co-attention fusion module to reduce feature redundancy and achieve effective multimodal data fusion. Furthermore, we design a loss function to handle missing modality issues and align cross-modal features. These components collectively harness multimodal data to boost network performance. Experimental results on the ADNI1 and ADNI2 datasets show that our method significantly surpasses existing unimodal and other multimodal models. Our code is available at https://github.com/justinhxy/ITFC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。