arXiv:2410.23937cs.LGcs.DS2024-10NeurIPS被引 2

提出新算法,可在非均匀设计下实现鲁棒稀疏回归,误差优于现有方法。

Robust Sparse Regression with Non-Isotropic Designs

  • 基于平方和松弛与加权Huber损失,通过过滤协变量提升鲁棒性。
  • 样本量 n ≥ Õ(k²/ε) 时误差为 O(√ε),n ≥ Õ(k⁴/ε³) 时误差可降至 o(√ε)。
  • 适用于重尾分布和对抗噪声,对高维稀疏问题有显著优势。

我们提出一种高效可计算的估计方法,用于在存在两种对手(静默对手与自适应对手)时的稀疏线性回归。设计了多个鲁棒算法,在静默对手仅添加高斯噪声的特殊情况下也优于现有最优方法。特别地,当样本数 n ≥ Õ(k²/ε) 且设计分布 D 的三阶与四阶矩有界时,存在多项式时间算法以高概率恢复信号,误差为 O(√ε)。此前,在高斯设计与噪声情况下,尚无已知多项式时间算法能在 n < d² 的稀疏设置中实现误差 o(√ε)。我们证明:若 D 的四阶与八阶矩满足一定条件,则存在多项式时间算法在 n ≥ Õ(k⁴/ε³) 时达到误差 o(√ε);对高斯分布,该误差为 O(ε^{3/4})。此外,对于所有对数凹分布,当 ε ≤ 1/polylog(d) 时,算法误差亦为 o(√ε)。算法核心为基于平方和松弛的协变量过滤与带 ℓ₁ 正则的加权 Huber 损失最小化。我们提出了适用于重尾设计与双对手场景的新分析框架。同时,我们通过统计查询下界补充了算法结果,表明所提估计器的样本复杂度可能接近最优。

原文摘要 · Abstract (English)

We develop a technique to design efficiently computable estimators for sparse linear regression in the simultaneous presence of two adversaries: oblivious and adaptive. We design several robust algorithms that outperform the state of the art even in the special case when oblivious adversary simply adds Gaussian noise. In particular, we provide a polynomial-time algorithm that with high probability recovers the signal up to error $O(\sqrt{\varepsilon})$ as long as the number of samples $n \ge \tilde{O}(k^2/\varepsilon)$, only assuming some bounds on the third and the fourth moments of the distribution ${D}$ of the design. In addition, prior to this work, even in the special case of Gaussian design and noise, no polynomial time algorithm was known to achieve error $o(\sqrt{\varepsilon})$ in the sparse setting $n < d^2$. We show that under some assumptions on the fourth and the eighth moments of ${D}$, there is a polynomial-time algorithm that achieves error $o(\sqrt{\varepsilon})$ as long as $n \ge \tilde{O}(k^4 / \varepsilon^3)$. For Gaussian distribution, this algorithm achieves error $O(\varepsilon^{3/4})$. Moreover, our algorithm achieves error $o(\sqrt{\varepsilon})$ for all log-concave distributions if $\varepsilon \le 1/\text{polylog(d)}$. Our algorithms are based on the filtering of the covariates that uses sum-of-squares relaxations, and weighted Huber loss minimization with $\ell_1$ regularizer. We provide a novel analysis of weighted penalized Huber loss that is suitable for heavy-tailed designs in the presence of two adversaries. Furthermore, we complement our algorithmic results with Statistical Query lower bounds, providing evidence that our estimators are likely to have nearly optimal sample complexity.

稀疏回归鲁棒学习高维统计对抗攻击

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。