让视觉语言模型的提示词更个性化，提升联邦学习效果。
Mixture of Experts Made Personalized: Federated Prompt Learning for Vision-Language Models
- 用多专家混合机制，让客户端下载多个预聚合提示词
- 本地注意力门控网络增强文本特征，更好对齐本地图像数据
- 在9个数据集上验证有效，适合资源受限的个性化联邦学习
联邦提示学习利用类似CLIP的视觉-语言模型（VLM）强大的表征学习能力，通过提示学习提升联邦学习性能。然而，现有方法多遵循传统联邦学习范式，客户端仅能从服务器下载单一全局聚合模型。这一范式虽适用于训练大型模型，但在轻量级提示词场景下并不合适。为此，本文提出个性化联邦多专家自适应提示（pFedMoAP），通过引入多个预聚合的固定非局部提示专家，使客户端能基于多专家混合机制实现提示学习的个性化。pFedMoAP采用本地注意力门控网络，动态生成增强文本特征，以更好地对齐本地图像数据，融合本地与下载的非局部自适应提示专家。在9个数据集、多种联邦设置下的大量实验表明该方法有效。代码已开源。
原文摘要 · Abstract (English)
Federated prompt learning benefits federated learning with CLIP-like Vision-Language Model's (VLM's) robust representation learning ability through prompt learning. However, current federated prompt learning methods are habitually restricted to the traditional FL paradigm, where the participating clients are generally only allowed to download a single globally aggregated model from the server. While justifiable for training full-sized models under federated settings, in this work, we argue that this paradigm is ill-suited for lightweight prompts. By facilitating the clients to download multiple pre-aggregated prompts as fixed non-local experts, we propose Personalized Federated Mixture of Adaptive Prompts (pFedMoAP), a novel FL framework that personalizes the prompt learning process through the lens of Mixture of Experts (MoE). pFedMoAP implements a local attention-based gating network that learns to generate enhanced text features for better alignment with local image data, benefiting from both local and downloaded non-local adaptive prompt experts. Extensive experiments on 9 datasets under various federated settings demonstrate the efficacy of the proposed pFedMoAP algorithm. The code is available at https://github.com/ljaiverson/pFedMoAP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。