arXiv:2411.10063cs.AIcs.CV2024-11被引 2

用提示学习实现安全高效的联邦域泛化,防止数据泄露。

Federated Domain Generalization via Prompt Learning and Aggregation

  • 用本地提示作为桥梁,在不共享原始数据的情况下传递知识。
  • 在四个基准数据集上均优于现有方法,提升模型在未知领域的泛化能力。
  • 仅训练少量提示和轻量聚合器,适合资源受限的联邦学习场景。

联邦域泛化(FedDG)旨在在保护隐私的前提下,通过应对数据异构性来提升全局模型在未见域中的泛化能力。现有方法通常在客户端间共享特定域的知识(如频谱信息、类别原型、数据风格),但这些知识直接从本地样本中提取,存在数据泄露风险,难以完全满足FedDG要求。本文提出一种基于提示学习的新型联邦域泛化框架——PLAN,利用预训练视觉语言模型(VLMs),通过本地学习提示作为更安全的知识传递媒介。PLAN包含两个训练阶段:第一阶段,各客户端使用自身数据进行文本与视觉提示学习,通过将全局提示作为共同参考,间接同步本地提示;第二阶段,客户端交换各自域特定的本地提示,并通过轻量级注意力聚合器选择性地融合为全局提示,最终用于适配VLM以应对未见目标域。该框架仅需训练少量提示和轻量聚合器,具备显著的计算与通信效率优势。大量实验表明,PLAN在四个基准数据集上均展现出优异的域泛化性能。

原文摘要 · Abstract (English)

Federated domain generalization (FedDG) aims to improve the global model generalization in unseen domains by addressing data heterogeneity under privacy-preserving constraints. A common strategy in existing FedDG studies involves sharing domain-specific knowledge among clients, such as spectrum information, class prototypes, and data styles. However, this knowledge is extracted directly from local client samples, and sharing such sensitive information poses a potential risk of data leakage, which might not fully meet the requirements of FedDG. In this paper, we introduce prompt learning to adapt pre-trained vision-language models (VLMs) in the FedDG scenario, and leverage locally learned prompts as a more secure bridge to facilitate knowledge transfer among clients. Specifically, we propose a novel FedDG framework through Prompt Learning and AggregatioN (PLAN), which comprises two training stages to collaboratively generate local prompts and global prompts at each federated round. First, each client performs both text and visual prompt learning using their own data, with local prompts indirectly synchronized by regarding the global prompts as a common reference. Second, all domain-specific local prompts are exchanged among clients and selectively aggregated into the global prompts using lightweight attention-based aggregators. The global prompts are finally applied to adapt VLMs to unseen target domains. As our PLAN framework requires training only a limited number of prompts and lightweight aggregators, it offers notable advantages in computational and communication efficiency for FedDG. Extensive experiments demonstrate the superior generalization ability of PLAN across four benchmark datasets.

联邦学习域泛化提示学习视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。