无需真实数据即可实现私密深度学习的高效预处理方法
DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning
- 用结构化噪声探测网络，无须私有或公开数据构造预处理矩阵
- 在ε≤3强隐私下性能超越DP-SGD，误差比公开数据方法低4.8%
- 适合医疗等数据受限的隐私敏感领域
差分隐私优化面临根本性几何不匹配：深度网络损失曲面高度各向异性，而DP-SGD注入各向同性噪声。二阶预处理可解决此问题，但估计曲率通常需消耗隐私预算的私有数据或引入分布偏移的公开数据。我们发现费雪信息矩阵可分解为可由合成噪声恢复的架构敏感项，以及可通过模态特异性频率统计近似的输入相关项。提出DP-KFC，通过结构化合成噪声探测网络构建KFAC预处理矩阵，无需私有或公开数据。实验表明，DP-KFC在多种模态下均持续优于DP-SGD和自适应基线，在ε≤3的强隐私条件下表现优异；其性能接近使用私有数据的预处理方法，而使用公开数据的方法性能下降最高达4.8%，证明了曲率可在不耗尽隐私预算且不引入分布偏移的情况下估计。该方法使医疗等监管严格领域的隐私保护学习成为可能。
原文摘要 · Abstract (English)
Differentially private optimization suffers from a fundamental geometric mismatch: deep networks have highly anisotropic loss landscapes, yet DP-SGD injects isotropic noise. Second-order preconditioning can resolve this, but estimating curvature typically requires private data (consuming privacy budget) or public data (introducing distribution shift). We show that the Fisher Information Matrix decouples into architectural sensitivity, recoverable via synthetic noise, and input correlations, approximable from modality-specific frequency statistics. We propose DP-KFC, which constructs KFAC preconditioners by probing networks with structured synthetic noise, requiring neither private nor public data. Empirically, DP-KFC consistently outperforms DP-SGD and adaptive baselines across diverse modalities in strong privacy regimes ($\varepsilon \leq 3$). DP-KFC matches private-data preconditioners while public-data variants degrade by up to $4.8\%$, showing that curvature can be estimated without consuming privacy budget or introducing distribution shift. This enables privacy-preserving learning in specialized domains (e.g., medical applications) where regulatory constraints make data scarce.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。