新算法同时自适应裁剪和动量,提升隐私训练效率。
DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
- 用同一组均值方差同时自适应裁剪和动量更新
- 无需手动调阈值,比基线模型更优的准确率与隐私保障
- 理论证明方差估计无偏,适合需要隐私保护的深度学习场景
差分隐私随机梯度下降(DP-SGD)是隐私保护机器学习的标准框架,但其依赖固定梯度裁剪阈值限制敏感度,存在实际局限。自适应裁剪算法如AdaClip通过调整裁剪前的梯度均值和方差来提升下降方向信息量,但未将这些统计量用于加速训练的动量。另一方面,DP-Adam利用梯度均值和方差进行类似Adam的动量更新以加速训练,但未用于自适应裁剪。本文提出一种新算法DP-MacAdam,将两者结合,使用相同的均值和方差估计同时完成裁剪和动量更新。我们证明该方法在无偏意义下估计梯度方差。实验表明,相比DP-SGD、AdaClip和DP-Adam基线,DP-MacAdam在不需手动调节裁剪阈值的前提下,实现了更高的模型效用。
原文摘要 · Abstract (English)
Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains a significant practical limitation. Adaptive clipping algorithms such as AdaClip shift and scale the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction. The shift and scaling parameters are selected adaptively based on the empirical mean and variance. However, in existing adaptive clipping algorithms, these empirical estimates have not been also used for momentum to accelerate training itself. On the other hand, DP-Adam is an algorithm that exploits Adam-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping. In this work, we propose Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum (DP-MacAdam), a novel algorithm that combines these two approaches so as to use the same mean and variance estimates for both clipping and momentum. We perform an analysis showing that DP-MacAdam estimates the gradient variances in a bias-free manner. In addition, we empirically evaluate the privacy and accuracy of DP-MacAdam, demonstrating that it achieves improved model utility compared to DP-SGD, AdaClip, and DP-Adam baselines, without requiring manual tuning of the clipping threshold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。