提出更优的私有自适应优化方法，提升隐私训练效果
On Design Principles for Private Adaptive Optimizers
- 先缩放再加噪，避免无偏二阶矩估计
- 在小规模语言模型上表现优于其他方法
- 理论与实证均支持其更适合实际应用
差分隐私训练中，对梯度添加球形噪声会损害自适应优化器（如AdaGrad、Adam）的性能。尽管已有多种改进算法，但现有研究多聚焦于简单任务和模型，结论可能无法推广。本文调研了多种变体，提供了更深入的理论理解，并进行了实证比较。结果表明，追求梯度二阶矩无偏估计的常见思路是错误的；相反，一种称为scale-then-privatize的简单方法（不保证无偏性）具有更优的理论性质，在小规模语言模型训练任务中表现最佳。此外，该方法使噪声添加更匹配相关噪声机制，后者在实践中更具优势。
原文摘要 · Abstract (English)
The spherical noise added to gradients in differentially private (DP) training undermines the performance of adaptive optimizers like AdaGrad and Adam, and hence many recent works have proposed algorithms to address this challenge. However, the empirical results in these works focus on simple tasks and models and the conclusions may not generalize to model training in practice. In this paper we survey several of these variants, and develop better theoretical intuition for them as well as perform empirical studies comparing them. We find that a common intuition of aiming for unbiased estimates of second moments of gradients in adaptive optimizers is misguided, and instead that a simple technique called scale-then-privatize (which does not achieve unbiased second moments) has more desirable theoretical behaviors and outperforms all other variants we study on a small-scale language model training task. We additionally argue that scale-then-privatize causes the noise addition to better match the application of correlated noise mechanisms which are more desirable to use in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。