通过局部模型融合加速联邦学习中预训练模型的适应
Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning
- 在客户端使用正则化模型插值增强本地训练
- 仅需少量通信轮次即可收敛到低损失区域
- 适合资源受限下快速适配大模型的联邦学习场景
联邦学习（FL）是一种利用分散数据协同训练模型的学习范式。近期研究表明，使用预训练权重初始化可有效提升模型性能。然而，当前预训练模型参数量大幅增加，导致其在联邦学习中适应所需的通信轮次显著增多。为缓解这一通信开销并提升预训练模型在联邦学习中的适应性能，我们提出一种基于模型插值的本地训练新方法——「局部优越汤」（Local Superior Soups）。该方法通过正则化模型插值，在少数通信轮次内促进客户端探索连通的低损失区域，从而成为预训练模型在联邦学习中无缝适配的催化剂。我们在多个广泛使用的联邦学习数据集上验证了该方法的有效性与高效性。代码已开源：https://github.com/ubc-tea/Local-Superior-Soups。
原文摘要 · Abstract (English)
Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.'' Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL. We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets. Our code is available at \href{https://github.com/ubc-tea/Local-Superior-Soups}{https://github.com/ubc-tea/Local-Superior-Soups}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。