通过谱记忆机制提升私有深度学习的精度与稳定性。
SMA-DP: Spectral Memory-Aware Differential Privacy for Deep Learning

- 引入谱感知记忆分支,利用历史私有更新增强梯度稳定性。
- 在CIFAR-100和CIFAR-10上优于或媲美现有私有优化基线。
- 支持可调隐私-效用权衡,适合对隐私保护要求高的场景。
差分隐私随机梯度下降(DP-SGD)通过逐样本裁剪和校准高斯噪声实现私有深度学习,但其高方差更新会降低在挑战性数据集上的性能。本文提出SMA-DP-SGD,一种基于谱记忆感知的差分隐私优化方法,通过仅使用先前私有化噪声释放构建分数阶记忆分支。受WeightWatcher启发的幂律谱指数提供分组可靠性信号,实验中按层实现,用于自适应调节衰减与有效记忆深度。私有历史对齐、范数匹配和暖启动激活稳定记忆贡献。隐私保持透明:在私有释放历史条件下,记忆分支固定,仅当前裁剪和缩放项依赖数据,比例系数β固定。因此保留清晰的条件敏感结构,当β=1时精确恢复分组差分隐私SGD。在CIFAR-100、CIFAR-10和MNIST上的实验表明,该方法在准确率上达到竞争性或更优表现,尤其在CIFAR-100和CIFAR-10上提升显著。CIFAR-10消融实验显示β控制隐私-效用轨迹,谱分析与记忆诊断证实有效记忆深度为短至中等,且记忆分支占比小。运行时分析表明该机制带来额外开销,约2.94倍于标准DP-SGD,揭示了自适应私有记忆与计算成本之间的实际权衡。
原文摘要 · Abstract (English)
Differentially private stochastic gradient descent (DP-SGD) enables private deep learning through per-example clipping and calibrated Gaussian noise, but its high-variance updates can reduce utility on challenging datasets. We propose \textbf{SMA-DP-SGD}, a \textbf{Spectral Memory-Aware Differentially Private Stochastic Gradient Descent} method that augments DP-SGD with a fractional memory branch built only from previously privatized noisy releases. WeightWatcher-inspired power-law spectral exponents provide group-wise reliability signals, instantiated layer-wise in our experiments, to adapt the decay and effective memory depth. Private-history alignment, norm matching, and warm-up activation stabilize the memory contribution. Privacy remains transparent: conditioned on the private release history, the memory branch is fixed, and the only newly data-dependent term is the current clipped sum scaled by a fixed coefficient \(β\). Hence, SMA-DP-SGD preserves a clean conditional sensitivity structure and exactly recovers group-wise DP-SGD when \(β=1\). Experiments on CIFAR-100, CIFAR-10, and MNIST show competitive or superior accuracy over several DP optimization baselines, with the largest gains on CIFAR-100 and CIFAR-10. CIFAR-10 ablations show that \(β\) controls the privacy--utility trajectory, while spectral and memory diagnostics confirm a controlled short-to-moderate effective memory depth and a small memory-branch ratio. Runtime analysis shows that the mechanism incurs additional overhead, about \(2.94\times\) DP-SGD in our CIFAR-10 implementation, revealing a practical trade-off between adaptive private memory and computational cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。