改进隐私优化器，让私有训练更准更快。
DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning
- 提出带偏差修正的私有版AdamW优化器
- 在文本分类上比现有方法高15%以上
- 适合需要强隐私保护的深度学习场景
随着深度学习广泛使用敏感数据，差分隐私（DP）为训练过程中的信息泄露提供了形式化保障。实现高效且保持隐私的DP优化器仍是重大挑战。近年来，AdamW因优异的实证表现成为主流选择。本文研究了DP-AdamW，并提出新版本DP-AdamW-BC，该版本对二阶矩估计器引入了差分隐私偏差修正。理论上分析了DP-AdamW与DP-AdamW-BC的隐私与收敛性。实验在多个隐私预算（ε=1, 3, 7）下评估两者性能。结果表明，DP-AdamW显著优于现有最优方案如DP-SGD、DP-Adam和DP-AdamBC：在文本分类任务上提升超15%，图像分类最高提升5%，图节点分类始终高出1%。此外，实验发现加入偏差修正（即DP-AdamW-BC）反而降低准确率，与DP-AdamBC提升效果相反。
原文摘要 · Abstract (English)
As deep learning methods increasingly utilize sensitive data on a widespread scale, differential privacy (DP) offers formal guarantees to protect against information leakage during model training. A significant challenge remains in implementing DP optimizers that retain strong performance while preserving privacy. Recent advances introduced ever more efficient optimizers, with AdamW being a popular choice for training deep learning models because of strong empirical performance. We study \emph{DP-AdamW} and introduce \emph{DP-AdamW-BC}, a differentially private variant of the AdamW optimizer with DP bias correction for the second moment estimator. We start by showing theoretical results for privacy and convergence guarantees of DP-AdamW and DP-AdamW-BC. Then, we empirically analyze the behavior of both optimizers across multiple privacy budgets ($ε= 1, 3, 7$). We find that DP-AdamW outperforms existing state-of-the-art differentially private optimizers like DP-SGD, DP-Adam, and DP-AdamBC, scoring over 15\% higher on text classification, up to 5\% higher on image classification, and consistently 1\% higher on graph node classification. Moreover, we empirically show that incorporating bias correction in DP-AdamW (DP-AdamW-BC) consistently decreases accuracy, in contrast to the improvement of DP-AdamBC improvement over DP-Adam.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。