研究ReLU神经网络梯度下降的隐式偏好，发现高维数据下逼近最小范数解。
How Does the ReLU Activation Affect the Implicit Bias of Gradient Descent on High-dimensional Neural Network Regression?
- 通过新奇的对偶分析追踪预测与系数演化，揭示激活模式快速稳定。
- 在高维随机特征下，隐式偏差与最小ℓ₂范数解差距为Θ(√(n/||λ||₁))。
- 适用于关注优化动态与泛化机制的机器学习研究者。
过参数化机器学习模型（如神经网络）通常导致训练目标欠定，存在多个全局最优解。隐式偏倚指常见优化算法（如梯度下降）收敛到的特定全局最优解。本文研究浅层ReLU模型在平方损失下，于高维随机特征上的梯度下降隐式偏倚。已有工作表明，在最坏情况下隐式偏倚不存在，或在严格正交数据下恰好对应最小ℓ₂-范数插值解。本文填补两者之间空白，证明对于足够高维的随机数据，梯度下降的隐式偏倚以高概率逼近最小ℓ₂-范数解，其差距为Θ(√(n/||λ||₁))，其中n为样本数，λ为数据协方差矩阵的谱。该结果基于一种新颖的原-对偶分析，精细追踪了预测值、数据空间系数及其相互作用的演化过程，并发现ReLU激活模式在随机数据上快速稳定。
原文摘要 · Abstract (English)
Overparameterized ML models, including neural networks, typically induce underdetermined training objectives with multiple global minima. The implicit bias refers to the limiting global minimum that is attained by a common optimization algorithm, such as gradient descent (GD). In this paper, we characterize the implicit bias of GD for training a shallow ReLU model with the squared loss on high-dimensional random features. Prior work (Vardi and Shamir, 2021) showed that the implicit bias does not exist in the worst-case, or corresponds exactly to the minimum-$\ell_2$-norm interpolating solution under exactly orthogonal data (Boursier et al., 2022). Our work interpolates between these two extremes and shows that, for sufficiently high-dimensional random data, the implicit bias approximates the minimum-$\ell_2$-norm solution with high probability with a gap on the order $Θ(\sqrt{n/||λ||_1})$, where $n$ is the number of training examples and $λ$ denotes the spectrum of the data covariance matrix. Our results are obtained through a novel primal-dual analysis that carefully tracks the evolution of predictions, data-span coefficients, as well as their interactions, and show that the ReLU activation pattern quickly stabilizes with high probability over random data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。