arXiv:2504.00411cs.LG2025-04

提出一种新型前向学习隐私保护方法,实现与传统方法相当的隐私保障。

Forward Learning with Differential Privacy

  • 基于前向传播扰动设计新算法,通过拒绝采样实现隐私保护
  • 动态调节噪声水平,训练中保持相近的隐私损失上限
  • 适用于对隐私要求高且需高效训练的深度学习场景

深度学习中的差分隐私(DP)是保障训练数据机密性的重要手段,同时需维持模型性能。现有DP训练方法通过裁剪并注入外部噪声到反向传播计算的样本梯度来提供隐私保障。不同于反向传播,基于扰动的前向学习算法在前向传播中天然引入噪声,并利用随机性估计梯度。尽管这些算法本身未显式提供隐私保护,但前向过程中的噪声引入为模型参数及其梯度提供了内在随机性保护,暗示其具备自然差分隐私潜力。本文提出一种私有化前向学习算法——差分私有统一似然比(DP-ULR),并证明其差分隐私保证。DP-ULR采用新颖的带拒绝机制的批量采样操作,结合经典差分隐私机制进行理论分析;同时引入理论指导的隐私控制器,动态调整每一步训练的噪声水平以控制隐私成本。实验表明,与基于反向传播的传统DP训练算法相比,DP-ULR在性能上具有竞争力,且维持了相近的隐私损失上限。

原文摘要 · Abstract (English)

Differential privacy (DP) in deep learning is a critical concern as it ensures the confidentiality of training data while maintaining model utility. Existing DP training algorithms provide privacy guarantees by clipping and then injecting external noise into sample gradients computed by the backpropagation algorithm. Different from backpropagation, forward-learning algorithms based on perturbation inherently add noise during the forward pass and utilize randomness to estimate the gradients. Although these algorithms are non-privatized, the introduction of noise during the forward pass indirectly provides internal randomness protection to the model parameters and their gradients, suggesting the potential for naturally providing differential privacy. In this paper, we propose a \blue{privatized} forward-learning algorithm, Differential Private Unified Likelihood Ratio (DP-ULR), and demonstrate its differential privacy guarantees. DP-ULR features a novel batch sampling operation with rejection, of which we provide theoretical analysis in conjunction with classic differential privacy mechanisms. DP-ULR is also underpinned by a theoretically guided privacy controller that dynamically adjusts noise levels to manage privacy costs in each training step. Our experiments indicate that DP-ULR achieves competitive performance compared to traditional differential privacy training algorithms based on backpropagation, maintaining nearly the same privacy loss limits.

差分隐私前向学习深度学习隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。