arXiv:2510.22880cs.LGcs.AI2025-10NeurIPS被引 6

解决多模态联邦学习中数据缺失与模态不全问题,让模型自动适配客户端差异。

Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data

  • 通过可学习的客户端嵌入控制,动态调整本地特征表示以匹配数据缺失模式。
  • 在严重数据缺失下性能提升最高达36.45%,显著优于现有方法。
  • 适合医疗、智能设备等存在数据不完整和模态异构的真实场景使用。

现实世界中的多模态联邦学习常面临客户端间数据不完整且异构的问题,导致局部特征表示错位,影响模型聚合效果。现有工作或假设模态集合不同但无缺失特征,或共享模态集但存在缺失特征。本文提出更通用且贴近实际的设定:每个客户端观测到不同的模态子集,且每种模态内部可能有缺失输入。为此,我们设计了一种新框架,采用基于可学习客户端嵌入控制的局部自适应表示,编码各客户端的数据缺失模式。这些嵌入作为重构信号,使全局聚合表示能对齐客户端本地上下文,更有效利用共享信息。此外,具有相似缺失模式的客户端嵌入可算法聚合,增强重构信号鲁棒性。在多个包含多样化缺失模式的联邦多模态基准上验证,该方法在极端数据缺失下实现最高36.45%的性能提升。理论分析提供了明确性能界,与实验结果一致。代码已开源。

原文摘要 · Abstract (English)

Multimodal federated learning in real-world settings often encounters incomplete and heterogeneous data across clients. This results in misaligned local feature representations that limit the effectiveness of model aggregation. Unlike prior work that assumes either differing modality sets without missing input features or a shared modality set with missing features across clients, we consider a more general and realistic setting where each client observes a different subset of modalities and might also have missing input features within each modality. To address the resulting misalignment in learned representations, we propose a new federated learning framework featuring locally adaptive representations based on learnable client-side embedding controls that encode each client's data-missing patterns. These embeddings serve as reconfiguration signals that align the globally aggregated representation with each client's local context, enabling more effective use of shared information. Furthermore, the embedding controls can be algorithmically aggregated across clients with similar data-missing patterns to enhance the robustness of reconfiguration signals in adapting the global representation. Empirical results on multiple federated multimodal benchmarks with diverse data-missing patterns across clients demonstrate the efficacy of the proposed method, achieving up to 36.45\% performance improvement under severe data incompleteness. The method is also supported by a theoretical analysis with an explicit performance bound that matches our empirical observations. Our source codes are provided at https://github.com/nmduonggg/PEPSY

联邦学习多模态数据缺失自适应表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。