固定梯度裁剪下实现差分隐私的高效收敛，突破传统限制。
Differentially Private Clipped-SGD: High-Probability Convergence with Arbitrary Clipping Level
- 采用固定裁剪阈值，适配标准差分隐私机制。
- 在重尾噪声下，收敛速度优于已有方法。
- 平衡隐私与收敛性，适合大模型训练场景。
梯度裁剪是深度学习中的基础工具，可提升随机一阶优化方法（如 SGD、AdaGrad、Adam）在重尾噪声下的高概率收敛性，而这类噪声在大规模语言模型训练中普遍存在。它也是差分隐私（DP）机制的关键组件。然而，现有高概率收敛分析通常要求裁剪阈值随优化步数增加，这与标准的高斯机制等差分隐私方案不兼容。本文首次为固定裁剪水平下的差分隐私裁剪 SGD（DP-Clipped-SGD）提供了高概率收敛分析，适用于凸与非凸光滑优化问题，在中央 α 阶矩有界的重尾噪声假设下（α∈(1,2]）。结果表明，使用固定裁剪水平时，算法能以更快的速率收敛至最优解邻域，并可调节该邻域大小以权衡隐私噪声与收敛速度，实现更精细的隐私-精度折衷。
原文摘要 · Abstract (English)
Gradient clipping is a fundamental tool in Deep Learning, improving the high-probability convergence of stochastic first-order methods like SGD, AdaGrad, and Adam under heavy-tailed noise, which is common in training large language models. It is also a crucial component of Differential Privacy (DP) mechanisms. However, existing high-probability convergence analyses typically require the clipping threshold to increase with the number of optimization steps, which is incompatible with standard DP mechanisms like the Gaussian mechanism. In this work, we close this gap by providing the first high-probability convergence analysis for DP-Clipped-SGD with a fixed clipping level, applicable to both convex and non-convex smooth optimization under heavy-tailed noise, characterized by a bounded central $α$-th moment assumption, $α\in (1,2]$. Our results show that, with a fixed clipping level, the method converges to a neighborhood of the optimal solution with a faster rate than the existing ones. The neighborhood can be balanced against the noise introduced by DP, providing a refined trade-off between convergence speed and privacy guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。