在标签噪声和群体分布偏移下,学习鲁棒的单层神经元。
Robust Learning of a Group DRO Neuron
- 通过群体分布鲁棒优化,寻找最坏情况下的最佳参数。
- 算法在任意标签噪声下仍保持常数倍性能竞争力。
- 适合需要抗干扰能力的机器学习预训练任务。
我们研究在标准平方损失下,面对任意标签噪声和群体级分布偏移时,学习单个神经元的问题。目标是找到一个最优参数 $oldsymbol{w}_*$,使其在最不利的群体加权组合下表现良好。具体地,我们解决一个群体分布鲁棒优化(Group DRO)问题:给定 $K$ 个独立分布 $oldsymbol{p}_{[1]},oldsymbol{p}_{[2]},oldsymbol{p}_{[K]}$,寻找近似 $oldsymbol{w}_*$ 来最小化凸组合权重 $oldsymbol{oldsymbol{ u}} riangleq oldsymbol{ u} oldsymbol{ u}$,其中目标函数为 $oldsymbol{ u} = oldsymbol{ u} oldsymbol{ u}$。该框架直接应对损失函数的固有非凸性,在任意标签扰动和群体分布偏移下提供鲁棒学习保证。所提出的对偶外推算法在大语言模型预训练基准上展现出潜力。
原文摘要 · Abstract (English)
We study the problem of learning a single neuron under standard squared loss in the presence of arbitrary label noise and group-level distributional shifts, for a broad family of covariate distributions. Our goal is to identify a ''best-fit'' neuron parameterized by $\mathbf{w}_*$ that performs well under the most challenging reweighting of the groups. Specifically, we address a Group Distributionally Robust Optimization problem: given sample access to $K$ distinct distributions $\mathcal p_{[1]},\dots,\mathcal p_{[K]}$, we seek to approximate $\mathbf{w}_*$ that minimizes the worst-case objective over convex combinations of group distributions $\boldsymbolλ \in Δ_K$, where the objective is $\sum_{i \in [K]}λ_{[i]}\,\mathbb E_{(\mathbf x,y)\sim\mathcal p_{[i]}}(σ(\mathbf w\cdot\mathbf x)-y)^2 - νd_f(\boldsymbolλ,\frac{1}{K}\mathbf1)$ and $d_f$ is an $f$-divergence that imposes (optional) penalty on deviations from uniform group weights, scaled by a parameter $ν\geq 0$. We develop a computationally efficient primal-dual algorithm that outputs a vector $\widehat{\mathbf w}$ that is constant-factor competitive with $\mathbf{w}_*$ under the worst-case group weighting. Our analytical framework directly confronts the inherent nonconvexity of the loss function, providing robust learning guarantees in the face of arbitrary label corruptions and group-specific distributional shifts. The implementation of the dual extrapolation update motivated by our algorithmic framework shows promise on LLM pre-training benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。