在高维数据比例增长下,揭示私密学习算法的误差新规律。
Differentially Private Learning Beyond the Classical Dimensionality Regime
- 提出比例维度范式,分析样本数与维度同步增长时的私密学习性能。
- 发现目标扰动算法存在类似双下降的训练误差现象,且不同算法表现互有优劣。
- 引入统计物理新工具,实现对隐私算法误差的精准刻画,适合研究者参考。
我们首次研究比例维度范式下的差分隐私学习,即当样本数 $n$ 和问题维度 $d$ 以恒定比例趋于无穷时($d/n \to δ$,$δ \in (0,\infty)$),该设定远超以往高维隐私学习中 $δ=0$ 或极小的假设。针对鲁棒线性回归与逻辑回归,我们给出了输出扰动、目标扰动和噪声随机梯度下降等经典算法的精确误差估计,其精度达 $1+o(1)$,远超现有粗略分析在该区域的失效表现。基于这些估计,我们发现目标扰动在鲁棒线性回归中存在前所未见的“双下降”式训练误差行为;并识别出输出扰动平均优于目标扰动的情形,反之亦然,表明二者相对性能比以往认知更复杂。为证明核心结论,我们引入了现代高斯比较不等式与源自统计物理的普遍性定律等全新概率工具。
原文摘要 · Abstract (English)
We initiate the study of differentially private learning in the proportional dimensionality regime, in which the number of data samples $n$ and problem dimension $d$ approach infinity at rates proportional to one another, meaning that $d/n\toδ$ as $n\to\infty$ for an arbitrary, given constant $δ\in(0,\infty)$. This setting is significantly more challenging than that of all prior theoretical work in high-dimensional differentially private learning, which, despite the name, has assumed that $δ= 0$ or is sufficiently small for problems of sample complexity $O(d)$, a regime typically considered "low-dimensional" or "classical" by modern standards in high-dimensional statistics. We provide sharp theoretical estimates of the error of several well-studied differentially private algorithms for robust linear regression and logistic regression, including output perturbation, objective perturbation, and noisy stochastic gradient descent, in the proportional dimensionality regime. The $1+o(1)$ factor precision of our error estimates enables a far more nuanced understanding of the price of privacy of these algorithms than that afforded by existing, coarser analyses, which are essentially vacuous in the regime we consider. Using our estimates, we discover a previously unobserved "double descent"-like phenomenon in the training error of objective perturbation for robust linear regression. We also identify settings in which output perturbation outperforms objective perturbation on average, and vice versa, demonstrating that the relative performance of these algorithms is less clear-cut than suggested by prior work. To prove our main theorems, we introduce several probabilistic tools that have not previously been used to analyze differentially private learning algorithms, such as a modern Gaussian comparison inequality and recent universality laws with origins in statistical physics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。