针对重尾数据的高维稀疏线性回归,提出两种差分隐私方法。
Differentially Private Sparse Linear Regression with Heavy-tailed Responses
- 用Huber损失与私有迭代硬阈值结合,处理重尾数据
- 误差界在样本量和维度下表现良好,且对尾部参数不敏感
- 适合隐私保护下的高维重尾数据建模,如金融、异常检测
作为机器学习与差分隐私中的基础问题,差分隐私线性回归已得到广泛研究。然而,现有方法主要关注常规数据分布或低维情形下的非规整数据。本文系统研究高维设置下具有重尾响应的差分隐私稀疏线性回归。首先提出DP-IHT-H方法,利用Huber损失与私有迭代硬阈值,在$(\varepsilon, δ)$-DP模型下实现估计误差界为$\tilde{O}\biggl( s^{* \frac{1}{2}} \cdot \biggl(\frac{\log d}{n}\biggr)^{\frac{ζ}{1 + ζ}} + s^{* \frac{1 + 2ζ}{2 + 2ζ}} \cdot \biggl(\frac{\log^2 d}{n \varepsilon}\biggr)^{\frac{ζ}{1 + ζ}} \biggr)$,其中$n$为样本量,$d$为维度,$s^*$为参数稀疏度,$ζ\in (0, 1]$刻画数据尾部的厚重程度。其次提出DP-IHT-L,在响应额外假设下进一步提升误差界至$\tilde{O}\Bigl(\frac{(s^*)^{3/2} \log d}{n \varepsilon}\Bigr)$,该界不再依赖于$ζ$。实验在合成与真实数据集上验证了所提方法优于传统面向‘常规’数据的差分隐私算法。
原文摘要 · Abstract (English)
As a fundamental problem in machine learning and differential privacy (DP), DP linear regression has been extensively studied. However, most existing methods focus primarily on either regular data distributions or low-dimensional cases with irregular data. To address these limitations, this paper provides a comprehensive study of DP sparse linear regression with heavy-tailed responses in high-dimensional settings. In the first part, we introduce the DP-IHT-H method, which leverages the Huber loss and private iterative hard thresholding to achieve an estimation error bound of \( \tilde{O}\biggl( s^{* \frac{1 }{2}} \cdot \biggl(\frac{\log d}{n}\biggr)^{\fracζ{1 + ζ}} + s^{* \frac{1 + 2ζ}{2 + 2ζ}} \cdot \biggl(\frac{\log^2 d}{n \varepsilon}\biggr)^{\fracζ{1 + ζ}} \biggr) \) under the $(\varepsilon, δ)$-DP model, where $n$ is the sample size, $d$ is the dimensionality, $s^*$ is the sparsity of the parameter, and $ζ\in (0, 1]$ characterizes the tail heaviness of the data. In the second part, we propose DP-IHT-L, which further improves the error bound under additional assumptions on the response and achieves \( \tilde{O}\Bigl(\frac{(s^*)^{3/2} \log d}{n \varepsilon}\Bigr). \) Compared to the first result, this bound is independent of the tail parameter $ζ$. Finally, through experiments on synthetic and real-world datasets, we demonstrate that our methods outperform standard DP algorithms designed for ``regular'' data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。