用多模态自编码器让脑部MRI在输入缺失时仍能准确重建和分类。
MultiMAE for Brain MRIs: Robustness to Missing Inputs Using Multi-Modal Masked Autoencoder
- 将每种MRI序列视为独立模态，通过融合架构实现多任务重建。
- 在输入缺失情况下，分割任务Dice提升10.1，分类MCC提升0.46。
- 适合处理不完整医学影像数据，尤其适用于脑部MRI研究。
医学影像中常见输入序列缺失，给依赖完整数据的深度学习模型带来挑战。受MultiMAE启发，本文提出一种用于3D脑部MRI的多模态、多任务学习的掩码自编码器（MAE）框架。将每种MRI序列视为独立模态，采用类后融合的Transformer编码器整合多序列信息，并为每种模态配置独立解码器流以实现多任务重建。该预训练策略使模型在保持各模态丰富表征的同时，可通过跨序列推理应对缺失输入。结果得到一个灵活且可泛化的脑部MRI编码器，能从可用输入推断缺失序列，并适配多种下游任务。在分割与分类任务中，相较于MAE-ViT基线，在存在缺失输入时整体Dice分数提升10.1，MCC提升0.46。实验验证了该预训练策略的有效性，代码已开源。
原文摘要 · Abstract (English)
Missing input sequences are common in medical imaging data, posing a challenge for deep learning models reliant on complete input data. In this work, inspired by MultiMAE [2], we develop a masked autoencoder (MAE) paradigm for multi-modal, multi-task learning in 3D medical imaging with brain MRIs. Our method treats each MRI sequence as a separate input modality, leveraging a late-fusion-style transformer encoder to integrate multi-sequence information (multi-modal) and individual decoder streams for each modality for multi-task reconstruction. This pretraining strategy guides the model to learn rich representations per modality while also equipping it to handle missing inputs through cross-sequence reasoning. The result is a flexible and generalizable encoder for brain MRIs that infers missing sequences from available inputs and can be adapted to various downstream applications. We demonstrate the performance and robustness of our method against an MAE-ViT baseline in downstream segmentation and classification tasks, showing absolute improvement of $10.1$ overall Dice score and $0.46$ MCC over the baselines with missing input sequences. Our experiments demonstrate the strength of this pretraining strategy. The implementation is made available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。