模型能处理任意缺失的脑影像数据,提升诊断准确率。
BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability

- 用跨模态蒸馏和解剖引导的掩码策略,统一训练多模态脑影像。
- 在多种模态缺失情况下,对阿尔茨海默病分类准确率提升6.2%~7.0%。
- 适合临床真实场景中模态不全的脑影像分析,如早期筛查或随访。
临床诊断常遵循模态递增路径:先做常规结构影像(如MRI),再根据需要添加FLAIR或T2序列,最后才使用分子影像(如淀粉样蛋白-PET)。因此患者实际获取的影像模态组合往往不完整且异质。但现有多数AI模型假设输入模态固定。本文提出BrainAnytime,一个在34,899个3D脑影像(来自五个数据集)上预训练的统一框架,支持任意模态可用情况下的脑影像分析,涵盖多序列MRI与淀粉样蛋白-PET。单个模型可处理从单一T1到完整多模态的所有情况。预训练通过跨模态蒸馏(RCMD)学习MRI与PET之间的结构-分子对应关系,并利用解剖图谱引导的课程掩码(PACM)优先关注疾病易感解剖区域,全部基于共享的3D掩码自编码器(Multi-MAE3D)。在四个下游任务、五种临床相关模态设置下,BrainAnytime显著优于专用模型、缺失模态基线及大型脑MRI预训练基础模型。尤其在健康对照(CN) vs. 阿尔茨海默病(AD)、CN vs. 轻度认知障碍(MCI)分类中,相对基线分别提升6.2%和7.0%的平均准确率。代码已公开于https://github.com/SDH-Lab/BrainAnytime。
原文摘要 · Abstract (English)
Clinical diagnostic workups typically follow a modality escalation pathway: after initial clinical evaluation, clinicians begin with routine structural imaging (e.g., MRI), selectively add sequences such as FLAIR or T2 to refine the differential, and reserve molecular imaging (e.g., amyloid-PET) for cases that remain uncertain after standard evaluation. Consequently, patients are observed with heterogeneous and often incomplete modality subsets. However, most current AI models assume fixed data modalities as the model inputs. In this paper, we present BrainAnytime, a unified pretraining framework pretrained on 34,899 3D brain scans from five datasets that support brain image analysis under arbitrary modality availability spanning multi-sequence MRI and amyloid-PET. A single model accepts whatever imaging is available, from a lone T1 scan to a full multimodal workup. Pretraining learns structural-molecular correspondences between MRI and PET via cross-modal distillation (RCMD) and prioritizes disease-vulnerable anatomy via atlas-guided curriculum masking (PACM), all within a shared 3D masked autoencoder (Multi-MAE3D). Across four downstream tasks and five clinically motivated modality settings, BrainAnytime largely outperforms modality-specific models, missing-modality baselines, and large-scale brain MRI pretrained foundation models on most modality settings. Notably, it surpasses the strongest missing-modality baselines with relative improvements of 6.2% and 7.0% in average accuracy on CN vs. AD and CN vs. MCI classification, respectively. Code is available at https://github.com/SDH-Lab/BrainAnytime.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。