arXiv:2504.21063cs.LGcs.AI2025-04被引 2

用词元级路由实现高效联邦域泛化,仅传1K参数

Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization

  • 词元级专家分配,让图像内不同部分匹配专属提示
  • 每轮通信仅需1000参数,远低于传统方法
  • 无需参数路由,适合资源受限的联邦学习场景

联邦域泛化(FedDG)旨在从数据异构的分布式客户端中学习全局泛化模型,同时保护隐私。现有方法通过学习单一全局提示来适配视觉语言模型(VLMs),但难以处理个性化样本。虽然混合专家(MoE)可实现专业化,但现有方法存在图像级专家分配粗粒度和参数化路由通信开销大的问题。为此,本文提出TRIP:一种词元级提示混合与无参数路由框架,将多个提示视为独立专家。不同于传统的图像级路由,TRIP将图像内不同词元分配给特定专家,并通过词元聚类与最优传输实现无参数路由。通过加权聚合各专家(权重为分配词元数)生成实例专属提示。此外,利用VLM零样本泛化能力设计无偏学习策略。在四个基准上实验表明,TRIP实现最优泛化性能,每轮通信仅需1000参数。

原文摘要 · Abstract (English)

Federated domain generalization (FedDG) aims to learn a globally generalizable model from decentralized clients with heterogeneous data while preserving privacy. Recent studies have introduced prompt learning to adapt vision-language models (VLMs) in FedDG by learning a single global prompt. However, such a one-prompt-fits-all learning paradigm typically leads to performance degradation on personalized samples. Although the mixture of experts (MoE) offers a promising solution for specialization, existing MoE-based methods suffer from coarse image-level expert assignment and high communication costs from parameterized routers. To address these limitations, we propose TRIP, a Token-level prompt mixture with parameter-free routing framework for FedDG, which treats multiple prompts as distinct experts. Unlike existing image-level routing designs, TRIP assigns different tokens within an image to specific experts. To ensure communication efficiency, TRIP incorporates a parameter-free routing mechanism based on token clustering and optimal transport. The instance-specific prompt is then synthesized by aggregating experts, weighted by the number of tokens assigned to each. Additionally, TRIP develops an unbiased learning strategy for prompt experts, leveraging the VLM's zero-shot generalization capability. Extensive experiments across four benchmarks demonstrate that TRIP achieves optimal generalization results, with communication of only 1K parameters per round. Our code is available at https://github.com/GongShuai8210/TRIP.

联邦学习提示学习模型压缩多专家

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。