用模型曲率提升隐私训练中的噪声相关性，显著提高准确率。
Correlating Cross-Iteration Noise for DP-SGD using Model Curvature
- 利用公开无标签数据估计模型曲率，优化跨迭代噪声相关性。
- 在多个数据集和隐私参数下，准确率明显优于现有方法。
- 适合关注隐私训练性能提升的研究者与实践者。
差分隐私随机梯度下降（DP-SGD）能够缓解深度学习训练中的诸多隐私风险，但其与普通SGD训练之间仍存在较大的准确率差距。为此，研究者提出了多种改进路径，其中一种称为DP-MF的方法通过关联不同迭代间的隐私噪声，使后续迭代可抵消前期噪声。本文提出一种名为NoiseCurve的新技术，利用从公开无标签数据中估算的模型曲率来提升跨迭代噪声相关性的质量。实验表明，在多个数据集、模型及隐私参数设置下，NoiseCurve计算出的噪声相关性相较于DP-MF方案能带来持续且显著的准确率提升。
原文摘要 · Abstract (English)
Differentially private stochastic gradient descent (DP-SGD) offers the promise of training deep learning models while mitigating many privacy risks. However, there is currently a large accuracy gap between DP-SGD and normal SGD training. This has resulted in different lines of research investigating orthogonal ways of improving privacy-preserving training. One such line of work, known as DP-MF, correlates the privacy noise across different iterations of stochastic gradient descent -- allowing later iterations to cancel out some of the noise added to earlier iterations. In this paper, we study how to improve this noise correlation. We propose a technique called NoiseCurve that uses model curvature, estimated from public unlabeled data, to improve the quality of this cross-iteration noise correlation. Our experiments on various datasets, models, and privacy parameters show that the noise correlations computed by NoiseCurve offer consistent and significant improvements in accuracy over the correlation scheme used by DP-MF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。