提出联邦预训练多模态大模型的新框架,解决数据隐私下的模型协同训练难题。
A Step Toward Federated Pretraining of Multimodal Large Language Models
- 冻结视觉编码器与语言模型,只联合训练跨模态投影器。
- 在多个数据集上超越基线,提升多模态理解性能。
- 适合需要保护数据隐私的医疗、金融等领域应用。
多模态大语言模型(MLLM)的发展受限于高质量公开数据的饱和,而大量多样化的多模态数据仍被保留在隐私敏感的孤岛中。联邦学习(FL)为解锁这些分布式资源提供了前景,但现有研究主要集中在微调阶段,预训练阶段几乎未被探索。本文首次提出联邦多模态对齐(Fed-MA)任务,一种轻量级预训练范式:冻结视觉编码器和语言模型,仅联合训练跨模态投影器。我们识别出两个关键挑战:(i) 局部投影器聚合时的参数干扰;(ii) 单次遍历协同SGD中的梯度振荡。为此,提出首个联邦MLLM预训练框架Fed-CMP。该框架采用典型可靠性感知聚合,构建公共空间将客户端投影器分解为共享对齐基和客户端特定系数,并进行可靠性加权融合以抑制干扰。同时引入正交性保持动量机制,通过正交投影对共享对齐基施加动量,保留几何结构的同时累积历史优化方向。基于公开数据集构建四个联邦预训练场景,大量实验表明,Fed-CMP显著优于现有基线。
原文摘要 · Abstract (English)
The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of diverse multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to unlock these distributed resources, but existing research focuses predominantly on fine-tuning, leaving the foundational pre-training phase largely unexplored. In this paper, we formally introduce the Federated MLLM Alignment (Fed-MA) task, a lightweight pre-training paradigm that freezes the vision encoder and LLM while collaboratively training the cross-modal projector. We identify two critical challenges in this setting: (i) parameter interference in aggregating local projectors; and (ii) gradient oscillations in one-pass collaborative SGD. To address these challenges, we propose Fed-CMP, a pioneering framework for federated MLLM pre-training. Fed-CMP employs Canonical Reliability-Aware Aggregation, which constructs a canonical space to decompose client projectors into a shared alignment basis and client-specific coefficients, then performs reliability-weighted fusion to suppress parameter interference. Furthermore, Fed-CMP introduces Orthogonality-Preserved Momentum, which applies momentum to the shared alignment basis via orthogonal projection, accumulating historical optimization directions while preserving geometric structure. We construct four federated pre-training scenarios based on public datasets, and extensive experiments validate that Fed-CMP significantly outperforms existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。