通过改进的ADMM方法,让联邦学习中部分模型个性化更稳定高效。
FedAPM: Federated Learning via ADMM with Partial Model Personalization
- 用一阶和二阶正则项构造增广拉格朗日函数,抑制客户端漂移
- 在四个异构多模态数据集上,测试准确率提升12.3%,收敛更快
- 适合数据差异大、需模型个性化的实际联邦学习场景
在联邦学习中,由于用户差异导致各设备数据非独立同分布,且存在多种数据模态,使用单一模型不切实际。通过部分模型个性化可有效应对这一问题,使部分参数在客户端间差异化,其余部分保持共享。然而我们发现,部分个性化可能加剧客户端漂移(本地模型与共享模型偏离),降低算法有效性与效率。为此,本文提出基于交替方向乘子法(ADMM)的联邦学习框架FedAPM,通过在目标函数中引入一阶与二阶近端项构建增广拉格朗日函数:二阶项提供固定修正,一阶项实现本地与共享模型间的补偿修正。理论分析表明,利用显式拉格朗日乘子估计,FedAPM在收敛性上更稳定高效。我们证明了从任意初始点出发,训练过程全局收敛至驻点,并在温和假设下达到常数、线性和次线性三种收敛速率。在四个异构多模态数据集上,采用不同评估指标验证性能,结果表明FedAPM实现更快更准的收敛,在测试准确率、F1分数和AUC上分别平均优于当前最优方法12.3%、16.4%和18.0%,且通信轮次更少。
原文摘要 · Abstract (English)
In federated learning (FL), the assumption that datasets from different devices are independent and identically distributed (i.i.d.) often does not hold due to user differences, and the presence of various data modalities across clients makes using a single model impractical. Personalizing certain parts of the model can effectively address these issues by allowing those parts to differ across clients, while the remaining parts serve as a shared model. However, we found that partial model personalization may exacerbate client drift (each client's local model diverges from the shared model), thereby reducing the effectiveness and efficiency of FL algorithms. We propose an FL framework based on the alternating direction method of multipliers (ADMM), referred to as FedAPM, to mitigate client drift. We construct the augmented Lagrangian function by incorporating first-order and second-order proximal terms into the objective, with the second-order term providing fixed correction and the first-order term offering compensatory correction between the local and shared models. Our analysis demonstrates that FedAPM, by using explicit estimates of the Lagrange multiplier, is more stable and efficient in terms of convergence compared to other FL frameworks. We establish the global convergence of FedAPM training from arbitrary initial points to a stationary point, achieving three types of rates: constant, linear, and sublinear, under mild assumptions. We conduct experiments using four heterogeneous and multimodal datasets with different metrics to validate the performance of FedAPM. Specifically, FedAPM achieves faster and more accurate convergence, outperforming the SOTA methods with average improvements of 12.3% in test accuracy, 16.4% in F1 score, and 18.0% in AUC while requiring fewer communication rounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。