arXiv:2501.06686cs.CRcs.LG2025-01

用随机微分方程建模神经网络,提升隐私保护能力。

Modeling Neural Networks with Privacy Using Neural Stochastic Differential Equations

  • 用神经随机微分方程(NSDE)约束模型学习过程,降低过拟合。
  • 在成员推理攻击下,NSDE的抗风险能力是ResNet的两倍。
  • 可直接替换传统网络,实现高隐私低代价的部署。

本文研究使用神经微分方程(NODE)建模具有内在隐私属性系统的可行性。与表达能力无限的前馈神经网络不同,NODE将学习限制在微分方程解的空间内。我们评估了其在成员推理攻击下的表现,发现相比ResNet等常规模型,节点具有两倍的抵抗能力。分析表明,有限表达性降低了对训练数据的过拟合。进一步地,我们通过理论和实证证明:神经随机微分方程(NSDE)是差分隐私(DP)学习器,提供与DP-SGD相同的可证明隐私保障。同时,NSDE在抵御成员推理攻击方面表现优异,隐私-效用权衡优于传统私有化方法。此外,我们提出一种即插即用策略,可高效将NSDE融入现有前馈架构以增强隐私性。

原文摘要 · Abstract (English)

In this work, we study the feasibility of using neural ordinary differential equations (NODEs) to model systems with intrinsic privacy properties. Unlike conventional feedforward neural networks, which have unlimited expressivity and can represent arbitrary mappings between inputs and outputs, NODEs constrain their learning to the solution of a system of differential equations. We first examine whether this constraint reduces memorization and, consequently, the membership inference risks associated with NODEs. We conduct a comprehensive evaluation of NODEs under membership inference attacks and show that they exhibit twice the resistance compared to conventional models such as ResNets. By analyzing the variance in membership risks across different NODE models, we find that their limited expressivity leads to reduced overfitting to the training data. We then demonstrate, both theoretically and empirically, that membership inference risks can be further mitigated by utilizing a stochastic variant of NODEs: neural stochastic differential equations (NSDEs). We show that NSDEs are differentially-private (DP) learners that provide the same provable privacy guarantees as DPSGD, the de-facto mechanism for training private models. NSDEs are also effective in mitigating membership inference attacks, achieving risk levels comparable to private models trained with DP-SGD while offering an improved privacyutility trade-off. Moreover, we propose a drop-in-replacement strategy that efficiently integrates NSDEs into conventional feedforward architectures to enhance their privacy.

隐私保护随机微分方程差分隐私模型安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。