提出新方法提升联邦多任务学习的表示能力
Toward Enhancing Representation Learning in Federated Multi-Task Settings
- 通过对比学习构建跨任务共享表示空间
- 在图像和语言任务上显著优于现有基线
- 适合处理模型与任务异构的现实场景
联邦多任务学习（FMTL）旨在为不同任务的用户提供定制化模型，同时保护数据隐私。现有方法通常假设用户间模型结构一致（即完全或部分同质），限制了其在真实场景中的应用。为此，本文不共享模型参数，而是学习跨任务的共享表示空间。提出 Muscle loss，一种新型对比学习目标，可同时对齐所有参与模型的表示。与传统成对对比方法不同，Muscle loss 的最小化等价于最大化所有模型表示间的互信息，能更有效捕捉任务间依赖关系。基于此，设计出 FedMuscle 算法，具备高效通信与天然处理模型及任务异构的能力。在多种图像与语言任务上的实验表明，该方法持续超越当前最优基线，在异构环境下表现稳健且有显著提升。
原文摘要 · Abstract (English)
Federated multi-task learning (FMTL) seeks to collaboratively train customized models for users with different tasks while preserving data privacy. Most existing approaches assume model congruity (i.e., the use of fully or partially homogeneous models) across users, which limits their applicability in realistic settings. To overcome this limitation, we aim to learn a shared representation space across tasks rather than shared model parameters. To this end, we propose Muscle loss, a novel contrastive learning objective that simultaneously aligns representations from all participating models. Unlike existing multi-view or multi-model contrastive methods, which typically align models pairwise, Muscle loss can effectively capture dependencies across tasks because its minimization is equivalent to the maximization of mutual information among all the models' representations. Building on this principle, we develop FedMuscle, a practical and communication-efficient FMTL algorithm that naturally handles both model and task heterogeneity. Experiments on diverse image and language tasks demonstrate that FedMuscle consistently outperforms state-of-the-art baselines, delivering substantial improvements and robust performance across heterogeneous settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。