arXiv:2504.20860cs.CV2025-04ICCV被引 5

FedMVP通过多模态上下文生成视觉提示,提升联邦学习中文本提示的泛化能力。

FedMVP: Federated Multimodal Visual Prompt Tuning for Vision-Language Models

  • 基于图像和类别属性特征动态生成多模态视觉提示。
  • 在20个数据集上对未见类别和域的泛化性能提升1.57%~2.26%。
  • 适合需保护隐私且要求模型泛化的联邦视觉语言任务。

在联邦学习中,文本提示调优通过在本地客户端数据上调整轻量级输入标记(或提示)来适应视觉-语言模型(如CLIP),同时保持网络权重冻结。训练后,仅提示由客户端共享至中心服务器进行聚合。然而,文本提示调优易过拟合已知概念,限制其对未见概念的泛化能力。为此,我们提出多模态视觉提示调优(FedMVP),将提示条件化于多模态上下文信息——来自输入图像和类别文本属性特征。FedMVP的核心是PromptFormer模块,通过交叉注意力机制协同对齐文本与视觉特征。动态生成的多模态视觉提示随后输入冻结的CLIP视觉编码器,并以CLIP相似性损失与一致性损失联合训练。在20个数据集、三个泛化设置下的广泛评估表明,FedMVP不仅保持了对分布内类别和领域的性能,还展现出更高的未见类别和领域泛化能力,超越现有最优方法1.57%~2.26%。代码开源于https://github.com/mainaksingha01/FedMVP。

原文摘要 · Abstract (English)

In federated learning, textual prompt tuning adapts Vision-Language Models (e.g., CLIP) by tuning lightweight input tokens (or prompts) on local client data, while keeping network weights frozen. After training, only the prompts are shared by the clients with the central server for aggregation. However, textual prompt tuning suffers from overfitting to known concepts, limiting its generalizability to unseen concepts. To address this limitation, we propose Multimodal Visual Prompt Tuning (FedMVP) that conditions the prompts on multimodal contextual information - derived from the input image and textual attribute features of a class. At the core of FedMVP is a PromptFormer module that synergistically aligns textual and visual features through a cross-attention mechanism. The dynamically generated multimodal visual prompts are then input to the frozen vision encoder of CLIP, and trained with a combination of CLIP similarity loss and a consistency loss. Extensive evaluation on 20 datasets, spanning three generalization settings, demonstrates that FedMVP not only preserves performance on in-distribution classes and domains, but also displays higher generalizability to unseen classes and domains, surpassing state-of-the-art methods by a notable margin of +1.57% - 2.26%. Code is available at https://github.com/mainaksingha01/FedMVP.

联邦学习多模态视觉语言模型提示调优

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。