arXiv:2606.26894cs.CV2026-06

提出3D多模态脑影像模型,显式建模模态内与跨模态关联。

Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI

论文配图:Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI
图 1 · 摘自论文原文
  • 设计四种注意力机制,分别捕捉模态内局部/全局特征与跨模态交互。
  • 在三个数据集上表现优于主流CNN与Transformer,多模态输入提升更显著。
  • 适合神经影像分析、医学图像理解等研究者参考。

脑部MRI对机器学习构成根本挑战:需从高维3D多模态数据中学习,但样本量有限,且解剖结构、病理状态和采集条件差异大。尽管多模态成像提供互补信息,有效融合仍困难。我们提出多模态内外上下文视觉变换器(MICViT),一种3D视觉变换器,显式建模模态特异性表示及跨模态的局部与全局交互。具体地,MICViT结合四种注意力机制:模态特定的局部与全局注意力用于模态内特征学习,跨模态局部与全局注意力捕获模态间交互。我们在三个异构数据集(UK Biobank, n=41,404;SOOP, n=1,062;Cam-CAN, n=613)上评估了MICViT,使用多种MRI模态(如T1、FLAIR、DWI、SWI)。MICViT在3D设置中持续优于最先进的CNN与Transformer基线。值得注意的是,其在多模态输入下收益更大,随模态增加性能提升更显著。结果表明,显式建模模态内与跨模态交互是释放多模态脑部MRI潜力的关键,为神经影像表征学习指明了新方向。

原文摘要 · Abstract (English)

Brain MRI poses a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities, despite the limited sample sizes typical of neuroimaging studies relative to the diversity in anatomy, pathology, and acquisition conditions. While multimodal imaging provides complementary information critical for clinical interpretation, effectively integrating these signals remains difficult. We propose Multimodal Intra- and Cross-Context Vision Transformer (MICViT), a 3D vision transformer that explicitly models both modality-specific representations and cross-modal interactions across local and global contexts. Concretely, MICViT combines four attention mechanisms: modality-specific local and global attention for intra-modal feature learning, and cross-modal local and global attention to capture interactions between modalities. We evaluate MICViT on brain age prediction across three heterogeneous datasets (UK Biobank, n=41,404; SOOP, n=1,062; Cam-CAN, n=613) using multiple MRI modalities (e.g. T1, FLAIR, DWI, SWI). MICViT consistently outperforms state-of-the-art CNN and transformer baselines in 3D settings. Notably, it benefits more strongly from multimodal inputs, yielding larger performance gains as additional modalities are incorporated. These results demonstrate that explicitly modeling intra- and cross-modal interactions is key to unlocking the full potential of multimodal brain MRI, highlighting a promising direction for representation learning in neuroimaging.

多模态脑影像3D视觉注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。