提出抗干扰的神经元学习算法，有效应对数据污染下的模型训练挑战。
Iterative thresholding for non-linear learning in the strong $\varepsilon$-contamination model
- 采用阈值化梯度下降，抵御标签与特征的恶意污染。
- 在$\ell_2$范数下达到$O(ν\sqrt{ε\log(1/ε)})$逼近误差，样本复杂度$O(d/ε)$。
- 适用于噪声敏感场景，适合高鲁棒性学习研究者参考。
研究在强ε-污染模型下，通过阈值化梯度下降学习单神经元模型的近似性能。假设数据满足 $y = σ(\mathbf{w}^{*} \cdot \mathbf{x}) + ξ$，其中 $σ$ 为非线性激活函数（如 sigmoid、leaky-ReLU、ReLU），噪声 $ξ$ 为高斯分布，协变量 $\mathbf{x}$ 来自子高斯分布。理论分析表明，该方法在 $\ell_2$-范数下达到 $O(ν\sqrt{ε\log(1/ε)})$ 的逼近界，样本复杂度为 $O(d/ε)$，失败概率为 $e^{-Ω(d)}$。针对线性回归（$σ(\mathbf{x}) = \mathbf{x}$）情形，逼近界提升至 $O(νε\log(1/ε))$，优于此前 $O(ν)$ 的结果。算法运行时间复杂度为 $O(\textrm{polylog}(N,d)\log(R/ε))$，当 $\|\mathbf{w}^{*}\|_2 \leq R$，显著优于 $O(\text{polylog}(N,d)/ε^2)$ 的现有方法。
原文摘要 · Abstract (English)
We derive approximation bounds for learning single neuron models using thresholded gradient descent when both the labels and the covariates are possibly corrupted adversarially. We assume the data follows the model $y = σ(\mathbf{w}^{*} \cdot \mathbf{x}) + ξ,$ where $σ$ is a nonlinear activation function, the noise $ξ$ is Gaussian, and the covariate vector $\mathbf{x}$ is sampled from a sub-Gaussian distribution. We study sigmoidal, leaky-ReLU, and ReLU activation functions and derive a $O(ν\sqrt{ε\log(1/ε)})$ approximation bound in $\ell_{2}$-norm, with sample complexity $O(d/ε)$ and failure probability $e^{-Ω(d)}$. We also study the linear regression problem, where $σ(\mathbf{x}) = \mathbf{x}$. We derive a $O(νε\log(1/ε))$ approximation bound, improving upon the previous $O(ν)$ approximation bounds for the gradient-descent based iterative thresholding algorithms of Bhatia et al. (NeurIPS 2015) and Shen and Sanghavi (ICML 2019). Our algorithm has a $O(\textrm{polylog}(N,d)\log(R/ε))$ runtime complexity when $\|\mathbf{w}^{*}\|_2 \leq R$, improving upon the $O(\text{polylog}(N,d)/ε^2)$ runtime complexity of Awasthi et al. (NeurIPS 2022).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。