提出无线联邦学习的二阶优化新方法，用高斯过程直接估算噪声下的全局海森矩阵。
GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning
- 基于高斯过程建模，从接收的噪声梯度中直接估计全局海森矩阵
- 在多个数据集上实现线性二次收敛，优于经典一阶和二阶方法
- 适合通信信道噪声大的无线联邦学习场景，尤其适用于资源受限设备
二阶优化方法能有效提升学习算法的收敛速度。在联邦学习（FL）中，这类方法需客户端向参数服务器（PS）共享本地海森矩阵，通信开销巨大。传统方案是仅通过一阶信息近似全局海森矩阵，但在无线空中传输（over-the-air FL）场景下，由于PS接收到的是带噪的局部梯度，该方法效果不佳。本文提出一种专为无线信道设计的新型二阶联邦学习框架。核心创新在于：PS利用非参数化方法，将未知海森矩阵建模为高斯过程，并结合梯度与海森矩阵的时间相关性及信道模型，构造出全局海森矩阵的随机估计器。该方法称为基于高斯过程的无线联邦学习海森建模（GP-FL），其具有线性二次收敛率。在多个数据集上的数值实验表明，GP-FL显著优于所有经典的基准一阶与二阶联邦学习方法。
原文摘要 · Abstract (English)
Second-order methods are widely adopted to improve the convergence rate of learning algorithms. In federated learning (FL), these methods require the clients to share their local Hessian matrices with the parameter server (PS), which comes at a prohibitive communication cost. A classical solution to this issue is to approximate the global Hessian matrix from the first-order information. Unlike in idealized networks, this solution does not perform effectively in over-the-air FL settings, where the PS receives noisy versions of the local gradients. This paper introduces a novel second-order FL framework tailored for wireless channels. The pivotal innovation lies in the PS's capability to directly estimate the global Hessian matrix from the received noisy local gradients via a non-parametric method: the PS models the unknown Hessian matrix as a Gaussian process, and then uses the temporal relation between the gradients and Hessian along with the channel model to find a stochastic estimator for the global Hessian matrix. We refer to this method as Gaussian process-based Hessian modeling for wireless FL (GP-FL) and show that it exhibits a linear-quadratic convergence rate. Numerical experiments on various datasets demonstrate that GP-FL outperforms all classical baseline first and second order FL approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。