arXiv:2601.15390cs.LG2026-01被引 1

提出轻量级联邦学习框架,让多模态大模型在保护隐私下高效训练。

FedUMM: A General Framework for Federated Learning with Unified Multimodal Models

  • 客户端用低秩适配器微调,服务器仅聚合适配器参数。
  • 16个客户端下性能仅轻微下降,仍媲美集中式训练。
  • 通信量降低超90%,适合医疗、金融等隐私敏感场景。

统一多模态模型(UMMs)作为可同时处理生成与理解任务的通用基础模型正在兴起。然而,这类模型通常在集中式环境下训练,需汇聚所有数据至中心服务器,限制了其在隐私敏感和地理分布场景的应用。本文提出FedUMM,一种适用于非独立同分布多模态数据、通信成本低的通用联邦学习框架。基于NVIDIA FLARE,FedUMM以BLIP3o为骨干,通过参数高效微调实现联邦化:客户端训练轻量级LoRA适配器并冻结主干模型,服务器仅聚合适配器更新。在VQA v2和GenEval组合生成基准上,采用狄利克雷控制的异构性设置,最多支持16个客户端。结果表明,随着客户端数量和异构性增加,性能略有下降,但仍保持与集中式训练相当的竞争力。进一步分析计算-通信权衡,证明仅传输适配器可使每轮通信量降低一个数量级以上,实现了实用化的联邦多模态模型训练。本工作为未来隐私保护的联邦统一多模态模型研究提供了实证经验。

原文摘要 · Abstract (English)

Unified multimodal models (UMMs) are emerging as strong foundation models that can do both generation and understanding tasks in a single architecture. However, they are typically trained in centralized settings where all training and downstream datasets are gathered in a central server, limiting the deployment in privacy-sensitive and geographically distributed scenarios. In this paper, we present FedUMM, a general federated learning framework for UMMs under non-IID multimodal data with low communication cost. Built on NVIDIA FLARE, FedUMM instantiates federation for a BLIP3o backbone via parameter-efficient fine-tuning: clients train lightweight LoRA adapters while freezing the foundation models, and the server aggregates only adapter updates. We evaluate on VQA v2 and the GenEval compositional generation benchmarks under Dirichlet-controlled heterogeneity with up to 16 clients. Results show slight degradation as client count and heterogeneity increase, while remaining competitive with centralized training. We further analyze computation--communication trade-offs and demonstrate that adapter-only federation reduces per-round communication by over an order of magnitude compared to full fine-tuning, enabling practical federated UMM training. This work provides empirical experience for future research on privacy-preserving federated unified multimodal models.

联邦学习多模态模型低通信隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。