用模块化组件让一个医学影像大模型同时处理多种任务和模态，提升诊断效果。
MAFM^3: Modular Adaptation of Foundation Models for Multi-Modal Medical AI
- 通过轻量级模块实现大模型在不同任务与模态间灵活切换。
- 在胸部CT上同时提升预后预测与分割性能，引入PET后Dice分数提升5%。
- 适合需要多任务、多模态医疗AI系统的研发团队使用。
基础模型在大规模数据上训练以捕捉领域通用规律，但在医学影像领域，数据稀缺使得为每个领域、模态或任务单独预训练困难。我们提出MAFM^3（多模态医学AI的基础模型模块化适配），一种框架，使单一基础模型可通过轻量级模块组件扩展至多样领域、任务与模态。这些模块作为专用技能，在推理时根据输入类型或临床目标灵活激活。不同于传统方法对每项新任务/模态孤立处理，MAFM^3提供统一且可扩展的高效多任务、多模态适配方案。实验验证：将初始用于分类的胸部CT基础模型适配为预后与分割模块，性能均提升；加入PET扫描后，Dice分数相较基线提升5%。结果表明，配备模块组件的基础模型能突破原始训练范围，演化为多任务、多模态系统。代码见https://github.com/Areeb2735/CTscan_prognosis_VLM。
原文摘要 · Abstract (English)
Foundational models are trained on extensive datasets to capture the general trends of a domain. However, in medical imaging, the scarcity of data makes pre-training for every domain, modality, or task challenging. Instead of building separate models, we propose MAFM^3 (Modular Adaptation of Foundation Models for Multi-Modal Medical AI), a framework that enables a single foundation model to expand into diverse domains, tasks, and modalities through lightweight modular components. These components serve as specialized skill sets that allow the system to flexibly activate the appropriate capability at the inference time, depending on the input type or clinical objective. Unlike conventional adaptation methods that treat each new task or modality in isolation, MAFM^3 provides a unified and expandable framework for efficient multitask and multimodality adaptation. Empirically, we validate our approach by adapting a chest CT foundation model initially trained for classification into prognosis and segmentation modules. Our results show improved performance on both tasks. Furthermore, by incorporating PET scans, MAFM^3 achieved an improvement in the Dice score 5% compared to the respective baselines. These findings establish that foundation models, when equipped with modular components, are not inherently constrained to their initial training scope but can evolve into multitask, multimodality systems for medical imaging. The code implementation of this work can be found at https://github.com/Areeb2735/CTscan_prognosis_VLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。