提出最优差分隐私ReLU回归方法,放宽数据假设。
Nearly Optimal Differentially Private ReLU Regression
- 设计单遍小批量算法DP-MBGLMtron,降低隐私预算和公共数据依赖。
- 理论证明误差下界为Ω(d²/(N²ε²)),算法逼近此下界。
- 适合关注隐私机器学习实用性的研究者与工程师。
本文研究差分隐私(DP)框架下的基础非凸学习问题——ReLU回归。以往工作依赖特征向量与标签范数有界的严格假设,本文将其放宽至数据可独立同分布采样自O(1)-次高斯分布的标准设定。当ε = Õ(√(1/N))且存在公共数据时,我们证明可实现上界为Õ(d²/(N²ε²))的过剩总体风险。进一步提出并分析了一种单遍小批量广义线性模型感知机算法(DP-MBGLMtron),降低了对ε值和公共数据的需求。利用追踪攻击论证技术,我们证明了(ε, δ)-DP算法估计误差的极小极大下界为Ω(d²/(N²ε²)),表明DP-MBGLMtron在对数因子内达到最优。实验结果支持理论发现。
原文摘要 · Abstract (English)
In this paper, we investigate one of the most fundamental nonconvex learning problems, ReLU regression, in the Differential Privacy (DP) model. Previous studies on private ReLU regression heavily rely on stringent assumptions, such as constant bounded norms for feature vectors and labels. We relax these assumptions to a more standard setting, where data can be i.i.d. sampled from $O(1)$-sub-Gaussian distributions. We first show that when $\varepsilon = \tilde{O}(\sqrt{\frac{1}{N}})$ and there is some public data, it is possible to achieve an upper bound of $\tilde{O}(\frac{d^2}{N^2 \varepsilon^2})$ for the excess population risk in $(ε, δ)$-DP, where $d$ is the dimension and $N$ is the number of data samples. Moreover, we relax the requirement of $ε$ and public data by proposing and analyzing a one-pass mini-batch Generalized Linear Model Perceptron algorithm (DP-MBGLMtron). Additionally, using the tracing attack argument technique, we demonstrate that the minimax rate of the estimation error for $(\varepsilon, δ)$-DP algorithms is lower bounded by $Ω(\frac{d^2}{N^2 \varepsilon^2})$. This shows that DP-MBGLMtron achieves the optimal utility bound up to logarithmic factors. Experiments further support our theoretical results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。