用二阶优化加速个性化联邦学习,减少通信轮次。
pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimization
- 基于梯度夹角设计个性化更新,融合客户端全局信息。
- 用正则化Fisher信息矩阵近似海森矩阵,实现快速收敛。
- 无需额外数据即可生成个性化模型,适合高异构场景。
个性化联邦学习(PFL)使客户端在数据高度异构下协同训练符合自身目标的个性化模型,克服传统联邦学习泛化能力差的问题。然而现有方法因依赖一阶优化导致收敛慢(线性收敛),且为寻找个性化模型需增加本地计算开销。二阶优化虽具二次收敛优势,但在PFL中受限于海森矩阵及其逆的计算难度。本文提出pFedSOP,通过Gompertz函数构建的局部与全局梯度夹角,生成个性化梯度更新,并以正则化Fisher信息矩阵(FIM)作为海森矩阵近似,实现高效二阶优化。该方法避免了在个性化搜索中引入额外数据,显著提升训练速度,减少通信轮次。在部分客户端参与、异构划分的图像分类数据集上,pFedSOP优于主流联邦学习与个性化联邦学习算法。
原文摘要 · Abstract (English)
Personalized Federated Learning (PFL) enables clients to collaboratively train personalized models tailored to their individual objectives, addressing the challenge of model generalization in traditional Federated Learning (FL) due to high data heterogeneity. However, existing PFL methods often require increased communication rounds to achieve the desired performance, primarily due to slow training caused by the use of first-order optimization, which has linear convergence. Additionally, many of these methods increase local computation because of the additional data fed into the model during the search for personalized local models. One promising solution to this slow training is second-order optimization, known for its quadratic convergence. However, employing it in PFL is challenging due to the Hessian matrix and its inverse. In this paper, we propose pFedSOP, which efficiently utilizes second-order optimization in PFL to accelerate the training of personalized models and enhance performance with fewer communication rounds. Our approach first computes a personalized local gradient update using the Gompertz function-based normalized angle between local and global gradient updates, incorporating client-specific global information. We then use a regularized Fisher Information Matrix (FIM), computed from this personalized gradient update, as an approximation of the Hessian to update the personalized models. This FIM-based second-order optimization speeds up training with fewer communication rounds by tackling the challenges with exact Hessian and avoids additional data being fed into the model during the search for personalized local models. Extensive experiments on heterogeneously partitioned image classification datasets with partial client participation demonstrate that pFedSOP outperforms state-of-the-art FL and PFL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。