arXiv:2509.22700cs.CV2025-09NeurIPS被引 4

提出新方法实现一次通信下的高效联邦提示学习,提升跨任务泛化能力。

Global Prompt Refinement with Non-Interfering Attention Masking for One-Shot Federated Learning

  • 设计非干扰注意力掩码机制,分离原始文本与可学习提示的交互
  • 在10个数据集上优于8种先进方法,类级与领域级泛化均提升
  • 适合资源受限场景下快速适应新任务的联邦学习应用

联邦提示学习(FPL)通过微调预训练模型上的轻量级提示实现高效通信。现有FPL方法依赖第二轮后才获得的全局信息进行协作,因而需多轮通信才能发挥优势。同时,现有单次通信方法通常仅适配已见任务,缺乏跨任务泛化能力。为此,本文提出全局提示优化与非干扰注意力掩码(GPR-NIAM)方法。核心是设计一种掩码机制,限制原始文本嵌入与可学习提示嵌入之间的过度交互。该方法由两个关键模块协同实现:首先,注意力隔离模块抑制可学习提示对原始文本的注意力,重加权反向注意力以保持跨任务泛化;其次,跨站点协同优化模块将分散的视觉知识整合至统一基础,并通过多源跨模态对齐校准全局提示,缓解数据异质性带来的不一致性。在10个基准数据集、两类任务上大量实验表明,GPR-NIAM在类级和领域级泛化上均超越8种先进方法。

原文摘要 · Abstract (English)

Federated Prompt Learning (FPL) enables communication-efficient adaptation by tuning lightweight prompts on top of frozen pre-trained models. Existing FPL methods typically rely on global information, which is only available after the second training round, to facilitate collaboration among client models. Therefore, they are inherently dependent on multi-round communication to fully exhibit their strengths. Moreover, existing one-shot federated learning methods typically focus on fitting seen tasks, but lack cross-task generalization. To bridge this gap, we propose the Global Prompt Refinement with Non-Interfering Attention Masking (GPR-NIAM) method for one-shot FPL. The core idea is to design a masking mechanism that restricts excessive interaction between the original text embeddings and the learnable prompt embeddings. GPR-NIAM achieves this through the collaboration of two key modules. Firstly, the attention isolation module suppresses attention from the learnable prompt tokens to the original text tokens, and reweights the reverse attention which preserves generalization across tasks. Secondly, the cross-silo collaborative refinement module integrates decentralized visual knowledge into a unified base and calibrates the global prompt through multi-source cross-modal knowledge alignment, further mitigating the inconsistency caused by data heterogeneity. Extensive experiments conducted on ten benchmark datasets under two tasks show that GPR-NIAM outperforms eight state-of-the-art methods in both class-level and domain-level generalization.

联邦学习提示学习跨任务泛化注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。