突破性证明非凸非光滑损失下噪声SGD的收敛隐私边界
Convergent Privacy Loss of Noisy-SGD without Convexity and Smoothness
- 通过霍尔德连续梯度条件，放宽了传统隐私分析的凸性和光滑性假设
- 在非凸非光滑场景下首次实现瑞尼隐私界随迭代次数收敛
- 适用于注重隐私保护的机器学习训练，尤其适合复杂模型场景
我们研究了在有界域上隐藏状态噪声SGD算法的差分隐私（DP）保证。标准的噪声SGD隐私分析假设所有内部状态均被披露，导致瑞尼隐私界随迭代次数发散。Ye & Shokri（2022）和Altschuler & Talwar（2022）证明了光滑（强）凸损失下的收敛边界，并提出了一个开放问题：这些假设能否被放松？本文给出了肯定答案——对非凸非光滑损失，只要损失函数具有霍尔德连续梯度，即可证明收敛的瑞尼隐私边界。此外，对于光滑强凸损失，我们的隐私边界严格优于现有最优结果。分析的核心在于多方面改进了偏移散度分析：包括前向沃瑟斯坦距离追踪、最优偏移分配识别，以及霍尔德降维引理。结果进一步揭示了隐藏状态分析在差分隐私中的优势及其适用范围。
原文摘要 · Abstract (English)
We study the Differential Privacy (DP) guarantee of hidden-state Noisy-SGD algorithms over a bounded domain. Standard privacy analysis for Noisy-SGD assumes all internal states are revealed, which leads to a divergent R'enyi DP bound with respect to the number of iterations. Ye & Shokri (2022) and Altschuler & Talwar (2022) proved convergent bounds for smooth (strongly) convex losses, and raise open questions about whether these assumptions can be relaxed. We provide positive answers by proving convergent R'enyi DP bound for non-convex non-smooth losses, where we show that requiring losses to have Hölder continuous gradient is sufficient. We also provide a strictly better privacy bound compared to state-of-the-art results for smooth strongly convex losses. Our analysis relies on the improvement of shifted divergence analysis in multiple aspects, including forward Wasserstein distance tracking, identifying the optimal shifts allocation, and the H"older reduction lemma. Our results further elucidate the benefit of hidden-state analysis for DP and its applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。