用分层专家模型提升脑部MRI的多模态问答能力,辅助神经外科手术规划。
Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRI
- 采用分层专家路由机制融合多序列3D MRI数据,实现高效端到端训练。
- 在多个数据集上平均性能领先基线5.5分,临床可用性提升46.6%。
- 首次构建3D脑MRI多模态问答数据集,适合医学AI与临床研究者参考。
多参数三维脑部MRI(mpMRI)在神经放射学中至关重要,但生成肿瘤位置、形态、大小及关键结构受累情况以支持神经外科规划仍具挑战。本文提出mpLLM,一种用于mpMRI视觉问答(VQA)的多模态大模型,可输出体积、形态、范围和粗略定位等临床可解释的肿瘤描述,作为对临床专家的补充。mpLLM采用提示驱动的分层混合专家(MoE)架构,通过模态级与标记级投影专家进行路由,融合多个3D序列,实现无需大规模图像-报告预训练的数据高效端到端训练。为缓解图像-文本配对监督不足问题,我们设计了一种基于专家分割标注生成临床相关问答的合成VQA协议,并经放射科医生协作验证。在多个mpMRI数据集上,mpLLM平均性能优于强基线5.5分(相对提升9.1%),放射科医生评估的临床可接受性提升15.9分(相对提升46.6%)。本研究主要贡献包括:(1)首个面向3D脑部mpMRI的VQA数据集;(2)用于跨相关3D序列联合推理的分层MoE架构;(3)基于专家支持的临床效用证据。代码已开源于https://github.com/arvindmvepa/mpllm,数据集将在发表后发布。
原文摘要 · Abstract (English)
Multiparametric 3D brain MRI (mpMRI) is central to neuroradiology, but producing tumor location, appearance, size, and involvement of critical structures for neurosurgical planning remains challenging. We introduce mpLLM, a multimodal LLM for visual question answering (VQA) on mpMRI that produces clinically interpretable tumor descriptors (e.g., volume, morphology, extent, and coarse localization) as an adjunct to clinical expertise for referring neurosurgeons. mpLLM uses a prompt-conditioned hierarchical mixture-of-experts (MoE) to fuse multiple 3D sequences via routing over modality- and token-level projection experts, enabling data-efficient end-to-end training without large-scale image-report pretraining. To address limited paired image-text supervision, we propose a synthetic VQA protocol that derives clinically grounded questions and answers from expert segmentation annotations and is validated with radiologist collaboration. Across multiple mpMRI datasets, mpLLM improves over strong medical VLM baselines by +5.5 points on average (+9.1% relative) and increases radiologist-rated clinical acceptability by +15.9 points (+46.6% relative). Our study features three main contributions: (1) the first VQA dataset for 3D brain mpMRI, (2) a hierarchical MoE architecture for joint reasoning over interrelated 3D sequences, and (3) expert-supported evidence of clinical utility. Source code is available at https://github.com/arvindmvepa/mpllm, and we will release the dataset upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。