FedMGP通过多组图文提示实现个性化联邦学习,提升模型在本地数据上的表现。
FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Prompts
- 每客户端使用多组图文提示,捕捉细粒度语义和实例特征。
- 动态聚合策略基于相似性采样,兼顾通用知识与本地特性的平衡。
- 参数高效设计,通信开销最小,适合资源受限的联邦场景。
本文提出FedMGP,一种面向视觉语言模型的个性化联邦提示学习新范式。每个客户端配备多组配对的文本与视觉提示,以捕获多样且细粒度的语义及实例级线索。引入多样性损失,促使各提示组专注于不同且互补的语义方面,确保整体覆盖更广泛的本地特征。通信阶段采用基于相似性引导的概率采样动态聚合策略:客户端计算其提示组与上轮全局提示的余弦相似度,通过softmax加权分布采样s组。该软选择机制优先聚合语义一致的知识,同时有效探索未充分表示的模式,平衡通用知识保留与客户端特异性。值得注意的是,FedMGP通过在多组间重新分配固定提示容量实现参数高效,相较所有联邦提示学习方法达到最优性能且通信参数最少。理论分析表明,动态聚合策略通过强化共享语义、抑制客户端噪声,促进鲁棒的全局表征学习。大量实验显示,FedMGP在多种联邦视觉语言基准上,持续优于现有方法,在个性化与领域泛化能力方面均表现卓越。代码将发布于https://github.com/weihao-bo/FedMGP.git。
原文摘要 · Abstract (English)
In this paper, we introduce FedMGP, a new paradigm for personalized federated prompt learning in vision-language models. FedMGP equips each client with multiple groups of paired textual and visual prompts, enabling the model to capture diverse, fine-grained semantic and instance-level cues. A diversity loss is introduced to drive each prompt group to specialize in distinct and complementary semantic aspects, ensuring that the groups collectively cover a broader range of local characteristics. During communication, FedMGP employs a dynamic prompt aggregation strategy based on similarity-guided probabilistic sampling: each client computes the cosine similarity between its prompt groups and the global prompts from the previous round, then samples s groups via a softmax-weighted distribution. This soft selection mechanism preferentially aggregates semantically aligned knowledge while still enabling exploration of underrepresented patterns effectively balancing the preservation of common knowledge with client-specific features. Notably, FedMGP maintains parameter efficiency by redistributing a fixed prompt capacity across multiple groups, achieving state-of-the-art performance with the lowest communication parameters among all federated prompt learning methods. Theoretical analysis shows that our dynamic aggregation strategy promotes robust global representation learning by reinforcing shared semantics while suppressing client-specific noise. Extensive experiments demonstrate that FedMGP consistently outperforms prior approaches in both personalization and domain generalization across diverse federated vision-language benchmarks. The code will be released on https://github.com/weihao-bo/FedMGP.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。