提出一种高效私密联邦学习优化方法,提升低隐私预算下的收敛速度。
DP-FedSOFIM: Differentially Private Federated Stochastic Optimization using Regularized Fisher Information Matrix
- 基于正则化费雪信息矩阵近似,用聚合梯度构建曲率信息
- 在不同隐私预算下,收敛更快且准确率优于多个基线方法
- 无需客户端计算海森矩阵,通信与内存开销小,适合大规模应用
差分隐私联邦学习(DP-FL)在严格隐私预算下常因噪声导致梯度质量下降而收敛缓慢。尽管二阶优化可加速训练,但现有方法存在显著扩展性问题:牛顿类方法需客户端计算海森矩阵,特征协方差方法随模型维度增长而性能下降。本文提出 DP-FedSOFIM,一种基于近似海森矩阵的简单可扩展二阶优化方法。该方法仅使用已隐私化的聚合梯度,在服务器端构建正则化代理费雪信息矩阵,捕捉有效曲率信息,无需完整海森矩阵计算或特征协方差估计。基于 Sherman-Morrison 公式的高效秩一更新机制,使通信成本与模型规模成正比,客户端内存需求仅为 O(d)。所有曲率与预处理操作均在服务器端对已隐私化梯度执行,不引入额外隐私开销。在 CIFAR-10 与 PathMNIST 数据集上的实验表明,DP-FedSOFIM 在多种隐私预算下均实现更快收敛,并持续优于多个竞争性私密联邦学习基线方法。
原文摘要 · Abstract (English)
Differentially private federated learning (DP-FL) often suffers from slow convergence under tight privacy budgets because the noise required for privacy preservation degrades gradient quality. Although second-order optimization can accelerate training, existing approaches for DP-FL face significant scalability limitations: Newton-type methods require clients to compute Hessians, while feature covariance methods scale poorly with model dimension. We propose DP-FedSOFIM, a simple and scalable Hessian approximation-based second-order optimization method for DP-FL. The method constructs a regularized proxy for the Fisher information matrix at the server using only privatized aggregated gradients, capturing useful curvature information without requiring full Hessian computations or feature covariance estimation. Efficient rank-one updates based on the Sherman-Morrison formula enable communication costs proportional to the model size and require only O(d) client-side memory. Because all curvature and preconditioning operations are performed at the server on already privatized gradients, DP-FedSOFIM introduces no additional privacy cost beyond the underlying privatized gradient release mechanism. Experiments on CIFAR-10 and PathMNIST demonstrate that DP-FedSOFIM converges faster and consistently achieves higher accuracy than several competitive differentially private federated learning baselines across a wide range of privacy budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。