私有优化中，噪声本身就能保证找到二阶驻点。
Noise is All You Need: Private Second-Order Convergence of Noisy SGD
- 利用隐私所需噪声实现二阶收敛，无需额外设计
- 在平滑假设下，非Lipschitz损失函数也能收敛到二阶驻点
- 适用于需要隐私保护的复杂优化场景
私有优化是机器学习中的重要课题，差分隐私随机梯度下降（DP-SGD）在理论与实践中均扮演关键角色。现有对DP-SGD的分析或依赖较强假设（如损失函数Lipschitz连续甚至凸性），或仅能证明一阶收敛（在非凸情形下可能停于鞍点）。尽管非私有版本的“噪声SGD”已实现二阶收敛，但更复杂的算法才能保证此性质。本文重新审视DP-SGD，发现：‘噪声即一切’——隐私所需的噪声本身，在标准平滑性假设下，即可自动带来二阶收敛，即使对于非Lipschitz损失函数亦然。因此，在最小假设下，现代私有优化的核心工具DP-SGD可几乎免费获得二阶收敛能力，直接用于寻找二阶驻点。
原文摘要 · Abstract (English)
Private optimization is a topic of major interest in machine learning, with differentially private stochastic gradient descent (DP-SGD) playing a key role in both theory and practice. Furthermore, DP-SGD is known to be a powerful tool in contexts beyond privacy, including robustness, machine unlearning, etc. Existing analyses of DP-SGD either make relatively strong assumptions (e.g., Lipschitz continuity of the loss function, or even convexity) or prove only first-order convergence (and thus might end at a saddle point in the non-convex setting). At the same time, there has been progress in proving second-order convergence of the non-private version of ``noisy SGD'', as well as progress in designing algorithms that are more complex than DP-SGD and do guarantee second-order convergence. We revisit DP-SGD and show that ``noise is all you need'': the noise necessary for privacy already implies second-order convergence under the standard smoothness assumptions, even for non-Lipschitz loss functions. Hence, we get second-order convergence essentially for free: DP-SGD, the workhorse of modern private optimization, under minimal assumptions can be used to find a second-order stationary point.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。