自适应剪裁与重要性梯度更新,提升高维神经网络的隐私保护性能。
AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks
- 基于参数重要性动态剪枝并自适应裁剪梯度,降低噪声影响。
- 在60%保留率下,CIFAR-10上达73.21%准确率,优于非私有基线。
- 适合对隐私和模型精度均有要求的深度学习场景。
差分隐私已被证明对随机梯度下降有效;然而,现有方法在高维设置中常因注入噪声规模随维度增加而性能下降。为此,我们提出AdaDPIGU——一种针对深度神经网络的新型差分隐私SGD框架,结合基于重要性的梯度更新机制。预训练阶段,采用差分隐私高斯机制估计各参数的重要性,同时保护隐私。梯度更新阶段,剔除低重要性坐标,并引入坐标级自适应裁剪机制,实现稀疏且噪声高效的梯度更新。理论上,我们证明了AdaDPIGU满足(ε, δ)-差分隐私并保持收敛性。大量实验验证其有效性,所有结果均在固定保留率60%下报告。在MNIST上,ε=8时测试准确率达99.12%,接近非私有模型。尤为显著的是,在CIFAR-10上ε=4时达到73.21%准确率,超过非私有基线的71.12%,表明自适应稀疏化可同时提升隐私与实用性。
原文摘要 · Abstract (English)
Differential privacy has been proven effective for stochastic gradient descent; however, existing methods often suffer from performance degradation in high-dimensional settings, as the scale of injected noise increases with dimensionality. To tackle this challenge, we propose AdaDPIGU--a new differentially private SGD framework with importance-based gradient updates tailored for deep neural networks. In the pretraining stage, we apply a differentially private Gaussian mechanism to estimate the importance of each parameter while preserving privacy. During the gradient update phase, we prune low-importance coordinates and introduce a coordinate-wise adaptive clipping mechanism, enabling sparse and noise-efficient gradient updates. Theoretically, we prove that AdaDPIGU satisfies $(\varepsilon, δ)$-differential privacy and retains convergence guarantees. Extensive experiments on standard benchmarks validate the effectiveness of AdaDPIGU. All results are reported under a fixed retention ratio of 60%. On MNIST, our method achieves a test accuracy of 99.12% under a privacy budget of $ε= 8$, nearly matching the non-private model. Remarkably, on CIFAR-10, it attains 73.21% accuracy at $ε= 4$, outperforming the non-private baseline of 71.12%, demonstrating that adaptive sparsification can enhance both privacy and utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。