arXiv:2510.25372cs.CVcs.LG2025-10中稿 · TMLR 2026被引 1

提出新方法,在联邦学习中实现视觉大模型的高效个性化与泛化。

Prompt Estimation from Prototypes for Federated Prompt Tuning of Vision Transformers

  • 用全局类原型和客户端先验自适应组合提示,实现每样本个性化。
  • 在多个数据异构场景下,性能超越现有最佳方法。
  • 适合资源受限的联邦学习场景,尤其适用于视觉大模型微调。

预训练视觉变换器(ViTs)的视觉提示微调(VPT)是一种高效的参数微调技术,能以极少参数适应下游任务,特别适合通信与计算受限的联邦学习(FL)。然而,全局提示微调难以跨异构客户端泛化,而个性化微调又易过拟合本地数据且缺乏泛化能力。本文提出 PEP-FedPT(基于原型的提示估计),统一实现泛化与个性化。该框架引入新型类上下文混合提示(CCMP),结合类特定提示与全局共享提示,通过全局类原型和客户端类先验动态加权生成每样本提示,无需存储客户端专属可训练参数。提示通过传统的联邦平均算法协同优化。在 CIFAR-100、TinyImageNet、DomainNet 和 iNaturalist 等数据集上的全面评估表明,PEP-FedPT 在多种数据异构场景下持续优于当前最优基线,为视觉变换器的高效且泛化的联邦提示微调奠定了坚实基础。

原文摘要 · Abstract (English)

Visual Prompt Tuning (VPT) of pre-trained Vision Transformers (ViTs) has proven highly effective as a parameter-efficient fine-tuning technique for adapting large models to downstream tasks with limited data. Its parameter efficiency makes it particularly suitable for Federated Learning (FL), where both communication and computation budgets are often constrained. However, global prompt tuning struggles to generalize across heterogeneous clients, while personalized tuning overfits to local data and lacks generalization. We propose PEP-FedPT (Prompt Estimation from Prototypes for Federated Prompt Tuning), a unified framework designed to achieve both generalization and personalization in federated prompt tuning of ViTs. Within this framework, we introduce the novel Class-Contextualized Mixed Prompt (CCMP) - based on class-specific prompts maintained alongside a globally shared prompt. For each input, CCMP adaptively combines class-specific prompts using weights derived from global class prototypes and client class priors. This approach enables per-sample prompt personalization without storing client-dependent trainable parameters. The prompts are collaboratively optimized via traditional federated averaging technique on the same. Comprehensive evaluations on CIFAR-100, TinyImageNet, DomainNet, and iNaturalist datasets demonstrate that PEP-FedPT consistently surpasses the state-of-the-art baselines under diverse data heterogeneity scenarios, establishing a strong foundation for efficient and generalizable federated prompt tuning of Vision Transformers.

联邦学习提示微调视觉模型高效微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。