解决多模态联邦学习中标签不统一与任务异构问题,实现自适应原型知识迁移。
Adaptive Prototype Knowledge Transfer for Federated Learning with Mixed Modalities and Heterogeneous Tasks
- 根据本地标签自适应选择原型构建方法,无需统一标签
- 在三个高度异构数据集上准确率提升0.42%~6.09%,召回率提升1.6%~3.89%
- 适合存在多模态、任务不一致的隐私保护联合建模场景
混合模态多模态联邦学习(MFL)允许多模态和单模态客户端在保护隐私的前提下协同训练模型。作为本地数据的代表性样本,原型以低资源消耗和无需先验知识的方式支持该场景。然而,现有基于原型的MFL方法假设客户端间标签统一且每个客户端任务相同,这在混合模态联邦学习中不现实。本文提出自适应原型多模态联邦学习(AproMFL)框架,通过自适应构建的原型实现无统一标签的知识迁移。客户端根据标签自适应选择原型构造方式;服务器将客户端原型转换为统一的多模态原型并聚类形成全局原型。针对任务异构导致的模型聚合问题,设计基于客户端关系图的动态加权聚合方案。进一步提出全局原型知识迁移损失和全局模型知识迁移损失,实现全局知识向本地知识的传递。实验表明,AproMFL在三个高度异构数据集(α=0.1)和两个异构任务上优于四个基线方法,最优准确率提升0.42%~6.09%,召回率提升1.6%~3.89%,显著优于基于FedAvg的FedIoT。
原文摘要 · Abstract (English)
Multimodal Federated Learning (MFL) with mixed modalities enables unimodal and multimodal clients to collaboratively train models while ensuring clients' privacy. As a representative sample of local data, prototypes offer an approach with low resource consumption and no reliance on prior knowledge for MFL with mixed modalities. However, existing prototype-based MFL methods assume unified labels across clients and identical tasks per client, which is impractical in MFL with mixed modalities. In this work, we propose an Adaptive prototype-based Multimodal Federated Learning (AproMFL) framework for mixed modalities to address the aforementioned issues. Our AproMFL transfers knowledge through adaptively-constructed prototypes without unified labels. Clients adaptively select prototype construction methods in line with labels; server converts client prototypes into unified multimodal prototypes and cluster them to form global prototypes. To address model aggregation issues in task heterogeneity, we develop a client relationship graph-based scheme to dynamically adjust aggregation weights. Furthermore, we propose a global prototype knowledge transfer loss and a global model knowledge transfer loss to enable the transfer of global knowledge to local knowledge. Experimental results show that AproMFL outperforms four baselines on three highly heterogeneous datasets ($α=0.1$) and two heterogeneous tasks, with the optimal results in accuracy and recall being 0.42%~6.09% and 1.6%~3.89% higher than those of FedIoT (FedAvg-based MFL), respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。