通过每样本动量与低通滤波，同时降低差分隐私训练中的噪声和偏差。
Enhancing DPSGD via Per-Sample Momentum and Low-Pass Filtering
- 用每样本动量平滑梯度，减少采样方差。
- 用低通滤波抑制高频噪声，不额外消耗隐私预算。
- 在保证严格差分隐私下，显著提升模型精度与隐私-效用平衡。
差分隐私随机梯度下降（DPSGD）被广泛用于训练具备正式隐私保障的深度神经网络。然而，引入差分隐私常因添加噪声和偏差而降低模型准确率。现有方法通常仅解决其中一端问题：减少隐私噪声可能加剧截断偏差，反之亦然。本文提出新方法DP-PMLF，结合每样本动量与低通滤波策略，同步缓解隐私噪声与截断偏差。该方法先使用每样本动量对梯度估计进行平滑，降低采样方差；再通过后处理低通滤波器衰减高频隐私噪声，不消耗额外隐私预算。我们提供了理论分析，证明在严格差分隐私保证下收敛速度更优。实验表明，相较于多个先进DPSGD变体，DP-PMLF显著提升了隐私-效用权衡性能。
原文摘要 · Abstract (English)
Differentially Private Stochastic Gradient Descent (DPSGD) is widely used to train deep neural networks with formal privacy guarantees. However, the addition of differential privacy (DP) often degrades model accuracy by introducing both noise and bias. Existing techniques typically address only one of these issues, as reducing DP noise can exacerbate clipping bias and vice-versa. In this paper, we propose a novel method, \emph{DP-PMLF}, which integrates per-sample momentum with a low-pass filtering strategy to simultaneously mitigate DP noise and clipping bias. Our approach uses per-sample momentum to smooth gradient estimates prior to clipping, thereby reducing sampling variance. It further employs a post-processing low-pass filter to attenuate high-frequency DP noise without consuming additional privacy budget. We provide a theoretical analysis demonstrating an improved convergence rate under rigorous DP guarantees, and our empirical evaluations reveal that DP-PMLF significantly enhances the privacy-utility trade-off compared to several state-of-the-art DPSGD variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。