arXiv:2502.13482cs.LGcs.CR2025-02被引 2

提出新型平滑归一化方法,实现高效隐私保护的分布式非凸优化。

Smoothed Normalization for Efficient Distributed Private Optimization

  • 用平滑归一化替代传统裁剪,减少梯度扰动对训练的影响。
  • 理论证明算法在隐私约束下仍能保持更快收敛速度。
  • 首次实现具有收敛保证的差分隐私分布式优化,适合联邦学习场景。

联邦学习可在保护参与者隐私的同时训练机器学习模型。然而,针对光滑非凸优化问题,尚无差分隐私的分布式方法。原因在于标准隐私技术需限制参与方贡献,通常通过更新裁剪实现。现有研究或忽略裁剪影响,或分析含裁剪的分布式算法但忽略隐私约束。本文提出一种基于单机设置中表现优异的平滑归一化方法,结合误差反馈机制,设计新分布式算法 α-NormEC。理论证明该方法收敛速率优于先前工作。将 α-NormEC 扩展至差分隐私设置后,获得首个具备可证明收敛性的差分隐私分布式优化算法。神经网络训练的实验结果表明,α-NormEC 在不同参数设置下均表现出稳健的收敛性。

原文摘要 · Abstract (English)

Federated learning enables training machine learning models while preserving the privacy of participants. Surprisingly, there is no differentially private distributed method for smooth, non-convex optimization problems. The reason is that standard privacy techniques require bounding the participants' contributions, usually enforced via $\textit{clipping}$ of the updates. Existing literature typically ignores the effect of clipping by assuming the boundedness of gradient norms or analyzes distributed algorithms with clipping but ignores DP constraints. In this work, we study an alternative approach via $\textit{smoothed normalization}$ of the updates motivated by its favorable performance in the single-node setting. By integrating smoothed normalization with an error-feedback mechanism, we design a new distributed algorithm $α$-$\sf NormEC$. We prove that our method achieves a superior convergence rate over prior works. By extending $α$-$\sf NormEC$ to the DP setting, we obtain the first differentially private distributed optimization algorithm with provable convergence guarantees. Finally, our empirical results from neural network training indicate robust convergence of $α$-$\sf NormEC$ across different parameter settings.

联邦学习差分隐私分布式优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。