动态调整私有化梯度裁剪与噪声,提升隐私学习准确率。
Dyn-D$^2$P: Dynamic Differentially Private Decentralized Learning with Provable Utility Guarantee
- 根据梯度收敛动态调节裁剪阈值和噪声强度
- 在强隐私保护下准确率显著优于固定噪声方法
- 首次提供动态隐私学习的可证明性能保证
现有去中心化差分隐私学习方法通常采用恒定梯度裁剪边界和固定水平的高斯噪声,导致模型准确率明显下降。本文提出一种面向一般时变有向网络的动态差分隐私去中心化学习方法(Dyn-D²P),利用高斯差分隐私(GDP)框架进行隐私预算管理,根据梯度收敛情况动态调整裁剪边界与噪声水平。该动态噪声策略在保持总隐私预算的前提下提升了模型准确率。大量实验表明,相较于使用固定噪声的方法,尤其在强隐私约束下,本方法表现更优。此外,本文首次为动态梯度裁剪与噪声的差分隐私非凸优化提供了可证明的模型效用界,其性能依赖于网络参数,关于节点数n的缩放因子为1/√n,误差项由梯度裁剪引入。
原文摘要 · Abstract (English)
Most existing decentralized learning methods with differential privacy (DP) guarantee rely on constant gradient clipping bounds and fixed-level DP Gaussian noises for each node throughout the training process, leading to a significant accuracy degradation compared to non-private counterparts. In this paper, we propose a new Dynamic Differentially Private Decentralized learning approach (termed Dyn-D$^2$P) tailored for general time-varying directed networks. Leveraging the Gaussian DP (GDP) framework for privacy accounting, Dyn-D$^2$P dynamically adjusts gradient clipping bounds and noise levels based on gradient convergence. This proposed dynamic noise strategy enables us to enhance model accuracy while preserving the total privacy budget. Extensive experiments on benchmark datasets demonstrate the superiority of Dyn-D$^2$P over its counterparts employing fixed-level noises, especially under strong privacy guarantees. Furthermore, we provide a provable utility bound for Dyn-D$^2$P that establishes an explicit dependency on network-related parameters, with a scaling factor of $1/\sqrt{n}$ in terms of the number of nodes $n$ up to a bias error term induced by gradient clipping. To our knowledge, this is the first model utility analysis for differentially private decentralized non-convex optimization with dynamic gradient clipping bounds and noise levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。