用低秩适配器混合实现高效联邦个性化，提升小数据场景泛化能力。
Collaborative and Efficient Personalization with Mixtures of Adaptors
- 通过客户端特异的低秩适配器混合实现分组个性化。
- 在数据稀缺时比全模型混合泛化更好，且优于单客户端微调。
- 内存高效、抗过拟合，适合资源受限的联邦学习场景。
现实中的联邦学习普遍存在异构数据。我们提出一种参数高效的框架FLoRAL，使客户端通过混合低秩适配器进行分组个性化，混合方式为客户端特异。FLoRAL是一种模型参数化方法，将个性化联邦学习建模为多任务学习问题，权重共享作为隐式正则项。该方法内存高效，因所有个性化参数（基础模型+适配器）均可联邦共享。实验表明，当数据稀缺时，FLoRAL的泛化能力优于全模型混合；且始终优于每个客户端独立微调适配器的模型。这验证了‘联邦个性化’的优势及其对过拟合的鲁棒性。我们推导了收敛速率，理论证明FLoRAL能有效降低基础模型梯度方差。
原文摘要 · Abstract (English)
Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model's gradients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。