用预训练模型增强联邦学习，解决数据少难题
AugFL: Augmenting Federated Learning with Pretrained Models
- 引入预训练模型作为知识源，通过正则化元学习提升个性化联邦学习
- 提出AugFL算法，通信效率高且无需暴露模型或增加本地计算负担
- 理论证明在非凸情况下仍能有效迁移知识，适合资源受限设备
联邦学习（FL）近年来受到广泛关注。然而，由于严格的隐私政策或物联网设备等参与方存储容量有限，实际去中心化学习环境中训练数据稀缺，制约了其有效部署。本文研究利用（大）预训练模型（PMs）——蕴含丰富通用/领域无关知识——来增强联邦学习，缓解从零开始训练所需的大量数据。具体地，我们构建由中央服务器和分布式客户端组成的网络化联邦学习系统。首先，将基于预训练模型的个性化联邦学习建模为基于正则化的联邦元学习问题，客户端协同学习一个元模型，该模型从服务器端私有的预训练模型中迁移知识。随后，设计一种基于不精确ADMM的算法AugFL，优化该问题时无需暴露预训练模型，也无需增加客户端额外计算开销。进一步，在一般非凸情况下建立了AugFL的理论保证，涵盖通信复杂度、适应性能及知识迁移收益。大量实验验证了AugFL相较于现有基线方法的有效性和优越性。
原文摘要 · Abstract (English)
Federated Learning (FL) has garnered widespread interest in recent years. However, owing to strict privacy policies or limited storage capacities of training participants such as IoT devices, its effective deployment is often impeded by the scarcity of training data in practical decentralized learning environments. In this paper, we study enhancing FL with the aid of (large) pre-trained models (PMs), that encapsulate wealthy general/domain-agnostic knowledge, to alleviate the data requirement in conducting FL from scratch. Specifically, we consider a networked FL system formed by a central server and distributed clients. First, we formulate the PM-aided personalized FL as a regularization-based federated meta-learning problem, where clients join forces to learn a meta-model with knowledge transferred from a private PM stored at the server. Then, we develop an inexact-ADMM-based algorithm, AugFL, to optimize the problem with no need to expose the PM or incur additional computational costs to local clients. Further, we establish theoretical guarantees for AugFL in terms of communication complexity, adaptation performance, and the benefit of knowledge transfer in general non-convex cases. Extensive experiments corroborate the efficacy and superiority of AugFL over existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。