解决医疗影像跨模态异构联邦学习难题,实现多模态医学影像统一建模。
FM$^2$: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

- 从零训练核心模型并引入专家混合模块,适应不同机构的影像模态差异。
- 在多任务基准上优于现有方法,跨模态泛化能力提升显著。
- 适合医疗数据隐私敏感场景下的多模态模型协同训练,如医院联盟研究。
构建医疗影像基础模型需跨机构汇聚数据,但隐私法规禁止集中式聚合。现有联邦基础模型或微调自然图像模型(医学迁移效果差),或仅在单一模态中从头训练,缺乏任务统一性。我们识别出关键挑战:影像模态异构性,即客户端存在重叠(共享模态但标签分布异构)与非重叠(各客户端模态完全独立)两种结构。提出FM²框架,从头训练主干网络以保持医学领域保真度,并可选集成生物医学预训练编码器实现视觉-语言对齐。每个客户端配备双专家混合模块(类别专属MoE用于个性化知识,域专属MoE用于共享跨模态表示),结合异构模态对齐(HMA)正则项,确保参数对齐,理论证明具有$O(1/ ext{√}T)$收敛与泛化保证。进一步引入基于本地保留的GPT-4o生成描述的增强学习(CEL),通过文本语义桥接实现跨模态表示迁移,并扩展至联邦医学VQA任务。在自建MIMH基准(分类与CEL)及真实世界医学VQA数据集上的实验表明,FM²在所有三类任务中持续优于最先进联邦基线,且具备强跨模态泛化能力。
原文摘要 · Abstract (English)
Building foundation models for medical imaging requires pooling data across institutions, yet privacy regulations prohibit centralized aggregation. Existing Federated Foundation Models either fine-tune natural-image models with poor medical-domain transfer, or train from scratch within a single modality, lacking the flexibility to unify tasks. We identify an under-explored challenge, Imaging Modality Heterogeneity, where clients operate under two structural regimes: Overlapped (shared modalities with heterogeneous label distributions) and Non-overlapped (fully disjoint modalities per client). We propose FM$^2$, a unified framework that trains the core backbone from scratch to preserve medical domain fidelity while optionally incorporating biomedical pretrained encoders for vision-language alignment. FM$^2$ equips each client with dual Mixture-of-Experts modules (a Class-wise MoE for personalized category knowledge and a Domain-wise MoE for shared cross-modality representations), coupled with a Heterogeneous Modality Alignment (HMA) regularizer that explicitly aligns modality-specific expert parameters, admitting provable $O(1/\sqrt{T})$ convergence and generalization guarantees. FM$^2$ further incorporates Caption-Enhanced Learning (CEL), where locally retained GPT-4o-generated captions serve as a textual semantic bridge enabling representation transfer across clients with disjoint modalities, and demonstrates extensibility to Federated Medical VQA. Experiments on our MIMH benchmark (classification and CEL) and real-world medical VQA datasets confirm consistent superiority over state-of-the-art federated baselines and strong out-of-modality generalization across all three tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。