arXiv:2607.21417cs.CV2026-07中稿 · ACM MM 2026

提出隐私保护的专家提示调优方法,更好平衡个性化与泛化。

Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach

论文配图:Towards Privacy-Preserving Federated Prompt Tuning under Data Heterogeneity: A Subspace-Decomposed Expert Approach
图 1 · 摘自论文原文
  • 用低秩共享因子+私有残差分解专家提示,压缩通信与噪声
  • 在11个异构数据集上,隐私约束下性能优于主流基线
  • 适合需要保护隐私且数据差异大的多设备协同场景

联邦提示调优(FPT)通过轻量级提示实现视觉-语言模型的协作适配。现有方法通常在本地差分隐私下采用分拆提示设计,结合共享提示进行全局迁移和私有提示进行本地适应。但单一共享提示可能过度平滑多样化的可迁移知识,削弱个性化与泛化间的平衡。多专家提示(MEPs)能更好捕捉这种多样性,但会扩大通信空间,增加差分隐私噪声和通信开销,且难以鲁棒地组合专家。本文提出FedSEPT,一种隐私保护的联邦子空间分解专家提示调优方法。具体地,采用子空间分解专家建模(SEM),通过共享低秩因子、固定公共基和私有残差参数化多个提示专家,将通信与差分隐私扰动限制在紧凑的因子空间内,并可在统一坐标系下直接聚合服务器端。进一步设计实例感知专家融合(IEF),通过设备端路由自适应组合语义互补的专家,并利用缓存的专家特异性文本特征实现高效的逐层融合。在11个异构基准上的大量实验表明,在相同隐私约束下,FedSEPT在本地适应与全局泛化之间取得了更优权衡。

原文摘要 · Abstract (English)

Federated prompt tuning (FPT) enables collaborative adaptation of vision--language models (VLMs) using lightweight prompts. Existing methods often address heterogeneity and privacy through a split-prompt design under local differential privacy (DP), combining a shared prompt for global transfer with private prompts for local adaptation. However, a single shared prompt may over-smooth diverse transferable knowledge, weakening the balance between personalization and generalization. Multi-expert prompts (MEPs) can better capture this diversity, but enlarge the communicated space, increasing DP noise and communication cost while making robust expert composition more difficult. We propose FedSEPT, a privacy-preserving Fed}erated Subspace-decomposed Expert Prompt Tuning. Specifically, we employ Subspace-decomposed Expert Modeling (SEM) to parameterize multiple prompt experts with shared low-rank factors, a fixed public basis, and private residuals, thereby confining communication and DP perturbation to a compact factor space while enabling direct server aggregation in a common coordinate system. We further design Instance-aware Expert Fusion (IEF), which adaptively combines semantically complementary experts via on-device routing and performs efficient logit-level fusion using cached expert-specific text features. Extensive experiments on 11 heterogeneous benchmarks show that, under the same privacy constraints, FedSEPT achieves a better trade-off between local adaptation and global generalization than strong baselines.

联邦学习提示调优隐私保护多专家

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。