提出一种无需存储历史噪声的高效差分隐私训练方法
DP-λCGD: Efficient Noise Correlation for Differentially Private Model Training
- 仅关联前一迭代的噪声并可控消减，降低内存开销
- 实验显示准确率优于标准DP-SGD，计算开销极小
- 适合对内存敏感的差分隐私模型训练场景
差分隐私随机梯度下降（DP-SGD）是实现机器学习模型形式化差分隐私保证的主流方法。近期一些扩展通过在训练迭代间引入相关噪声提升了精度。矩阵分解机制是其中典型，但需关联多个迭代的噪声并存储先前添加的噪声向量，导致显著内存开销。本文提出一种新噪声相关策略：仅与前一迭代噪声相关，并可控地消除部分噪声。该方法基于伪随机噪声生成器实现噪声重生成，无需存储历史噪声。因此，其内存开销与标准DP-SGD相同，无需额外存储。我们证明其计算开销极低，并在实验中验证了相比DP-SGD的精度提升。
原文摘要 · Abstract (English)
Differentially private stochastic gradient descent (DP-SGD) is the gold standard for training machine learning models with formal differential privacy guarantees. Several recent extensions improve its accuracy by introducing correlated noise across training iterations. Matrix factorization mechanisms are a prominent example, but they correlate noise across many iterations and require storing previously added noise vectors, leading to substantial memory overhead in some settings. In this work, we propose a new noise correlation strategy that correlates noise only with the immediately preceding iteration and cancels a controlled portion of it. Our method relies on noise regeneration using a pseudorandom noise generator, eliminating the need to store past noise. As a result, it requires no additional memory beyond standard DP-SGD. We show that the computational overhead is minimal and empirically demonstrate improved accuracy over DP-SGD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。