提出隐私保护的变分在线牛顿优化方法，提升敏感数据训练的稳定性。
DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton
- 用噪声修正的平方梯度估计曲率，减少隐私噪声与贝叶斯采样噪声的冲突。
- 在CIFAR-10上，ε较大时性能优于DP-SGD和DP-Adam，但强隐私下表现下降。
- 适合关注隐私保护下神经网络不确定性建模的研究者使用。
差分隐私为敏感数据上的神经网络训练提供正式隐私保障，而贝叶斯深度学习则提供了不确定性感知预测的严谨框架。两者的结合仍具挑战性，因隐私噪声可能与贝叶斯后验采样引入的随机性相互干扰。本文通过改进的变分在线牛顿（IVON）优化器，研究差分隐私的变分贝叶斯学习。提出DP-IVON-Gradsq，一种私有化版本的IVON。该方法利用噪声校正的平方梯度估计器构建曲率，从经过隐私保护的梯度中获取信息，降低后验采样噪声与隐私噪声的直接交互，同时保持了IVON的类Adam计算效率。在CIFAR-10上，对DP-SGD和DP-Adam在不同隐私预算（ε）下的对比实验表明：当ε较大（弱到中等隐私约束）时，DP-IVON-Gradsq具有竞争力；但在强隐私约束下性能下降。代码已公开于https://github.com/NourJamoussi/DP-IVON-Gradsq.git。
原文摘要 · Abstract (English)
Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of $\varepsilon$, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。