揭示高维下加噪梯度下降的隐私与效用动态关系
High-Dimensional Privacy-Utility Dynamics of Noisy Stochastic Gradient Descent on Least Squares
- 用连续扩散模型精确分析高维加噪SGD
- 无需梯度敏感度知识仍可实现隐私保护
- 适用于大规模带正则化最小二乘问题
优化与隐私的权衡已成为隐私保护机器学习的核心议题。加噪随机梯度下降（Noisy SGD）在大规模场景中已成为基石算法，通过在每一步更新中注入精心校准的噪声来实现差分隐私（differential privacy），即严格的隐私保障。以往研究主要提供统计风险和隐私损失的各类界，但在高维情形下，该过程的精确行为仍不明确。本文采用扩散方法对加噪SGD进行精确分析，从连续时间视角刻画了高维情形下统计风险演化与隐私损失动态。此外，我们研究了一种无需显式已知梯度敏感度的加噪SGD变体，不同于现有工作需通过梯度裁剪假设或施加敏感度。具体针对带有ℓ₂正则化的最小二乘问题。
原文摘要 · Abstract (English)
The interplay between optimization and privacy has become a central theme in privacy-preserving machine learning. Noisy stochastic gradient descent (SGD) has emerged as a cornerstone algorithm, particularly in large-scale settings. These variants of gradient methods inject carefully calibrated noise into each update to achieve differential privacy, the gold standard notion of rigorous privacy guarantees. Prior work primarily provides various bounds on statistical risk and privacy loss for noisy SGD, yet the \textit{exact} behavior of the process remains unclear, particularly in high-dimensional settings. This work leverages a diffusion approach to analyze noisy SGD precisely, providing a continuous-time perspective that captures both statistical risk evolution and privacy loss dynamics in high dimensions. Moreover, we study a variant of noisy SGD that does not require explicit knowledge of gradient sensitivity, unlike existing work that assumes or enforces sensitivity through gradient clipping. Specifically, we focus on the least squares problem with $\ell_2$ regularization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。