解决移动端多模态联邦学习中数据缺失问题,提升系统鲁棒性。
FedMobile: Enabling Knowledge Contribution-aware Multi-modal Federated Learning with Incomplete Modalities
- 跨节点共享特征重建缺失模态,避免单点插值。
- 支持90%模态信息缺失下仍保持性能,优于现有方法。
- 适合异构设备多模态数据融合场景,如智能健康监测。
网络物联网(WoT)增强了基于网络和普适计算平台间的互操作性,补充了现有物联网标准。多模态联邦学习(FL)被引入以通过融合多源移动传感数据来增强WoT,同时保护隐私。然而,移动传感系统中多模态FL面临的关键挑战是模态不完整,部分模态可能不可用或仅部分捕获,可能导致系统性能与可靠性下降。当前多模态FL框架通常训练多个单模态FL子系统,或在节点端使用插值技术近似缺失模态。这些方法忽略了不同节点间不完整模态的共享潜在特征空间,且无法区分低质量节点。为此,我们提出FedMobile,一种新的知识贡献感知多模态联邦学习框架,旨在应对缺失模态下的稳健学习。FedMobile优先考虑本地到全局的知识迁移,利用跨节点的多模态特征信息重构缺失特征。通过严格的节点贡献评估和知识贡献感知聚合规则,提升系统性能并增强对模态异质性的鲁棒性。在五个广泛认可的多模态基准数据集上的实证评估表明,即使高达90%的模态信息缺失,或两种模态数据随机缺失,FedMobile仍能维持稳健学习,优于现有最先进基线方法。
原文摘要 · Abstract (English)
The Web of Things (WoT) enhances interoperability across web-based and ubiquitous computing platforms while complementing existing IoT standards. The multimodal Federated Learning (FL) paradigm has been introduced to enhance WoT by enabling the fusion of multi-source mobile sensing data while preserving privacy. However, a key challenge in mobile sensing systems using multimodal FL is modality incompleteness, where some modalities may be unavailable or only partially captured, potentially degrading the system's performance and reliability. Current multimodal FL frameworks typically train multiple unimodal FL subsystems or apply interpolation techniques on the node side to approximate missing modalities. However, these approaches overlook the shared latent feature space among incomplete modalities across different nodes and fail to discriminate against low-quality nodes. To address this gap, we present FedMobile, a new knowledge contribution-aware multimodal FL framework designed for robust learning despite missing modalities. FedMobile prioritizes local-to-global knowledge transfer, leveraging cross-node multimodal feature information to reconstruct missing features. It also enhances system performance and resilience to modality heterogeneity through rigorous node contribution assessments and knowledge contribution-aware aggregation rules. Empirical evaluations on five widely recognized multimodal benchmark datasets demonstrate that FedMobile maintains robust learning even when up to 90% of modality information is missing or when data from two modalities are randomly missing, outperforming state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。