提出新方法IHM,让私有线性回归更准且更省数据。
Near-Optimal Private Linear Regression via Iterative Hessian Mixing
- 基于高斯压缩思路,迭代混合海森矩阵提升精度。
- 理论误差比现有最优方法降低约根号维度倍数。
- 适合对隐私与精度要求高的真实数据建模任务。
我们研究通过基于压缩的机制实现有界数据 $(X,Y)$ 下的差分隐私普通最小二乘法(DP-OLS)。尽管高斯压缩方法已被探索,但通常被认为不如直接扰动充分统计量 $(X^{ op}X, X^{ op}Y)$ 的自适应充分统计量扰动(AdaSSP)方法竞争力强。该方法被证明接近信息论最优,并具有优异的实证表现。本文提出迭代海森混合(IHM)算法,基于高斯压缩方法并受海森压缩启发。我们证明IH M满足差分隐私,并给出经验风险超出的保证。其边界优于AdaSSP,消除了可能高达数据维度平方根倍的乘性因子。IHM的设计基于对先前高斯压缩方法的新精度保证,明确了其适用场景并揭示了如何克服其固有局限。我们在大量数据集上进行严格实验,结果表明IH M始终优于现有基线,包括AdaSSP。
原文摘要 · Abstract (English)
We study differentially private ordinary least squares (DP-OLS) with bounded data $(X,Y)$ via sketching-based mechanisms. While Gaussian sketching approaches have been explored for DP-OLS \citep{sheffet2017differentially}, they are typically viewed as less competitive than the Adaptive Sufficient Statistics Perturbation (AdaSSP) method \citep{wang_adassp}, which directly perturbs the sufficient statistics $(X^{\top}X, X^{\top}Y)$. This method was shown to be close to information-theoretically optimal, while also exhibiting strong empirical performance. In this work, we propose the \emph{Iterative Hessian Mixing} (IHM), an algorithm that builds on Gaussian sketching approaches to DP-OLS and is inspired by the Iterative Hessian Sketch of \citet{pilanci_hessiansketch}. We prove that IHM is differentially private and provide utility guarantees in the form of excess empirical risk bounds. These bounds improve upon those of AdaSSP by removing a multiplicative factor that can be as large as the square root of the data dimension. The design of the IHM is based on new accuracy guarantees that we present for prior Gaussian sketching approaches for DP-OLS, which clarify when these methods are expected to perform well and how IHM circumvents their inherent limitations. We also conduct a rigorous empirical evaluation on a large suite of datasets, demonstrating that IHM consistently outperforms prior baselines, including AdaSSP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。