提出高效私有联邦学习算法,兼顾精度与通信成本。
Statistical Limits and Efficient Algorithms for Differentially Private Federated Learning

- 融合FedAvg初始化与FedSGD优化,降低偏差并减少通信轮次。
- 在客户端数量慢增长时,精度接近FedSGD但通信次数显著减少。
- 提供可量化隐私下的最优性基准,适用于医疗、金融等敏感数据场景。
联邦学习是一种在多个用户设备或数据库间协作训练机器学习模型的主流框架。本文研究差分隐私(DP)联邦M估计中估计精度、隐私约束与通信开销之间的权衡。文献中两种主流方法为:易受高聚合偏差影响的FedAvg,以及通信开销大的FedSGD。为提升精度并降低通信成本,我们提出FedHybrid,通过改进的FedAvg初始化启动FedSGD。进一步提出FedNewton,通过平均本地牛顿迭代来减少FedAvg的偏差,在客户端数量充分缓慢增长时,其估计精度接近FedSGD,但通信轮次大幅减少。我们建立了这些估计器的差分隐私版本在有限样本下的均方误差上界,其依赖于客户端数量、本地样本量、隐私预算及迭代次数。同时推导出任意迭代式私有联邦过程的极小极大下界,作为评估方法最优性差距的基准。我们在计算机视觉数据集MNIST和CIFAR-10上对逻辑回归与神经网络进行数值实验,验证了所提方法的有效性。
原文摘要 · Abstract (English)
Federated Learning is a leading framework for training ML and AI models collaboratively across numerous user devices or databases. We study the trade-offs among estimation accuracy, privacy constraints, and communication cost for differentially private (DP) federated M estimation. The two standard methods in the literature are FedAvg, which may suffer from high federation bias, and FedSGD, which can incur high communication cost. Aimed at improving accuracy at a reduced communication cost, we propose FedHybrid, which uses FedSGD starting with an improved initialization by the FedAvg estimator. We propose FedNewton, which averages local Newton iterations to reduce bias in FedAvg, achieving an estimation accuracy comparable to FedSGD with much fewer communication rounds when the number of clients grows sufficiently slowly. We establish finite sample upper bounds on the mean-squared error rates of the DP versions of these estimators as functions of the number of clients, local sample sizes, privacy budget, and number of iterations. We further derive a minimax lower bound on the MSE of any iterative private federated procedure that provides a benchmark to assess the optimality gap of these methods. We numerically evaluate our methods for training a logistic regression and a neural network on the computer vision datasets MNIST and CIFAR-10.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。