针对大模型联邦学习中的分布外泛化难题,提出轻量适配器方案提升个性化性能。
Federated Adapter on Foundation Models: An Out-Of-Distribution Approach
- 采用适配器实现参数高效微调,降低大规模模型训练开销。
- 通过特征距离正则化使客户端模型对未见任务保持稳定表现。
- 理论证明方法在非凸条件下收敛,适合数据异构的联邦场景。
随着基础模型日益重要,联邦基础模型(FedFM)成为在分布式数据下隐私保护协作微调模型的一种方法。但基础模型的多样性带来了分布外(OOD)泛化的挑战,即新任务或客户端出现分布偏移时性能下降。现有常规联邦学习的OOD方法难以适用,因模型参数量大、数据异构性强。为此,我们提出FedOA,采用基于适配器的参数高效微调,并引入基于特征距离的正则化,为每个客户端构建个性化适配器以对齐分布并保障OOD泛化能力。理论上,我们证明了传统联邦聚合的全局模型天然具备OOD泛化性,而我们的方法通过利用全局模型信息进行正则化,进一步提升个性化模型的泛化能力,且在一般非凸设定下可保证收敛。实验上,在多个自然语言处理任务的基准数据集上验证了方法的有效性。
原文摘要 · Abstract (English)
As foundation models gain prominence, Federated Foundation Models (FedFM) have emerged as a privacy-preserving approach to collaboratively fine-tune models in federated learning (FL) frameworks using distributed datasets across clients. A key challenge for FedFM, given the versatile nature of foundation models, is addressing out-of-distribution (OOD) generalization, where unseen tasks or clients may exhibit distribution shifts leading to suboptimal performance. Although numerous studies have explored OOD generalization in conventional FL, these methods are inadequate for FedFM due to the challenges posed by large parameter scales and increased data heterogeneity. To address these, we propose FedOA, which employs adapter-based parameter-efficient fine-tuning methods for efficacy and introduces personalized adapters with feature distance-based regularization to align distributions and guarantee OOD generalization for each client. Theoretically, we demonstrate that the conventional aggregated global model in FedFM inherently retains OOD generalization capabilities, and our proposed method enhances the personalized model's OOD generalization through regularization informed by the global model, with proven convergence under general non-convex settings. Empirically, the effectiveness of the proposed method is validated on benchmark datasets across various NLP tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。