提出可处理缺失模态的多模态MRI预训练方法，提升脑肿瘤分析泛化能力。
Multimodal Masked Autoencoder Pre-training for 3D MRI-Based Brain Tumor Analysis with Missing Modalities
- 设计统一模型，支持任意模态组合的预训练与微调
- 在多个下游任务上优于从头训练，且重建缺失模态效果好
- 适合临床中模态不全的实际场景，减少重复建模开销
多模态磁共振成像（MRI）是脑肿瘤临床诊疗的一线手段，对手术规划、治疗监测和生物标志物识别至关重要。预训练大模型能学习可迁移表征，减少标注依赖，尤其适用于标注稀缺的医学影像领域。然而，现有方法通常假设所有模态在预训练和微调阶段均完整可用。实际中，因采集问题、专家不可用或小规模内部数据集设计，缺失模态普遍存在。传统做法需为每种模态组合单独训练模型，资源消耗大且不适用于临床。为此，本文提出BM-MAE，一种专为多模态MRI设计的掩码图像建模预训练策略。同一预训练模型可无缝适应任意可用模态组合，提取融合模态内与模态间信息的丰富表征。微调时无需结构改动，仍可受益于全模态预训练。大量实验表明，该方法在多个下游任务上性能优于或媲美需为每种模态子集单独预训练的基线，显著优于从头训练。此外，模型还能高效重建缺失模态，凸显其实用性。代码与训练模型见：https://github.com/Lucas-rbnt/BM-MAE
原文摘要 · Abstract (English)
Multimodal magnetic resonance imaging (MRI) constitutes the first line of investigation for clinicians in the care of brain tumors, providing crucial insights for surgery planning, treatment monitoring, and biomarker identification. Pre-training on large datasets have been shown to help models learn transferable representations and adapt with minimal labeled data. This behavior is especially valuable in medical imaging, where annotations are often scarce. However, applying this paradigm to multimodal medical data introduces a challenge: most existing approaches assume that all imaging modalities are available during both pre-training and fine-tuning. In practice, missing modalities often occur due to acquisition issues, specialist unavailability, or specific experimental designs on small in-house datasets. Consequently, a common approach involves training a separate model for each desired modality combination, making the process both resource-intensive and impractical for clinical use. Therefore, we introduce BM-MAE, a masked image modeling pre-training strategy tailored for multimodal MRI data. The same pre-trained model seamlessly adapts to any combination of available modalities, extracting rich representations that capture both intra- and inter-modal information. This allows fine-tuning on any subset of modalities without requiring architectural changes, while still benefiting from a model pre-trained on the full set of modalities. Extensive experiments show that the proposed pre-training strategy outperforms or remains competitive with baselines that require separate pre-training for each modality subset, while substantially surpassing training from scratch on several downstream tasks. Additionally, it can quickly and efficiently reconstruct missing modalities, highlighting its practical value. Code and trained models are available at: https://github.com/Lucas-rbnt/BM-MAE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。