arXiv:2505.16329stat.MLcs.LG2025-05被引 1

提出最优隐私线性回归算法,适配高维数据与差分隐私需求。

High-Dimensional Private Linear Regression with Optimal Rates

  • 基于一阶梯度下降与自适应梯度裁剪,实现高维隐私保护
  • 风险达到 $O(γ + γ^2/ρ^2)$,且为理论最优
  • 适用于高维数据,尤其适合对隐私敏感的机器学习场景

本文研究在差分隐私(DP)约束下的高维线性回归问题,考虑样本数 $n$ 与维度 $d$ 比例趋于常数 $γ = d/n$ 的情形。针对满足 $ρ^2/2$ 零集中差分隐私的一次遍历梯度下降(DP-GD)算法,建立其轨迹的确定性等价方程组(常微分方程)。该框架揭示了小于典型样本梯度范数的梯度裁剪常数如何提升实际性能。对于条件良好的数据,通过合理选择裁剪常数与学习率,可实现非渐近风险 $O(γ + γ^2 / ρ^2)$,并证明该速率是极小极大最优的。对于协方差谱服从幂律分布的病态数据,风险呈现关于 $γ$ 的幂律缩放行为,其指数随隐私参数 $ρ$ 变化。分析表明,激进的梯度裁剪与递减学习率策略具有理论优势。

原文摘要 · Abstract (English)

Differentially private (DP) linear regression has received significant attention in the recent theoretical literature, with several approaches proposed to improve error rates. Our work considers the popular high-dimensional regime with random data, where the number of training samples $n$ and the input dimension $d$ grow at a proportional rate $d / n \to γ$, and it studies a family of one-pass DP gradient descent (DP-GD) algorithms satisfying $ρ^2 / 2$ zero concentrated DP. In this setting, we establish a deterministic equivalent for the DP-GD trajectory in terms of a system of ordinary differential equations. This allows to analyze the effect of gradient clipping constants that are smaller than the typical norm of the per-sample gradients - a setup shown to improve performance in practice. For well-conditioned data, we show that DP-GD, upon properly choosing clipping constant and learning rate, achieves the non-asymptotic risk of $O(γ+ γ^2 / ρ^2)$, and we establish that this rate is minimax optimal. Then, we consider the ill-conditioned case where the data covariance spectrum follows a power-law distribution, and we show that the risk displays a power-like scaling law in $γ$, highlighting the change in the exponent as a function of the privacy parameter $ρ$. Overall, our analysis demonstrates the benefits of practical algorithmic design choices, including aggressive gradient clipping and decaying learning rate schedules.

差分隐私线性回归高维统计优化算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。