将梯度下降视为感知机算法，揭示神经网络优化加速机制
Gradient Descent as a Perceptron Algorithm: Understanding Dynamics and Implicit Acceleration
- 将非线性模型的梯度下降等价为广义感知机算法，简化分析
- 证明两层模型迭代复杂度达˜O(√d)，远快于线性模型的Ω(d)
- 为神经网络隐式加速现象提供理论解释，适合优化研究者参考
即便在神经网络训练中广泛应用梯度下降（GD），其优化动态（如收敛速度、迭代轨迹、函数值振荡及隐式加速）仍难理解。本文针对逻辑损失下的非线性模型，证明GD步骤可简化为广义感知机算法（Rosenblatt, 1958），从而获得更简洁的分析框架。借助经典线性代数工具，我们在一个极简例子中证明：两层模型的非线性结构可使迭代复杂度达到˜O(√d)，显著优于线性模型的Ω(d)（d为特征数量）。该结果解释了神经网络中的优化动态与隐式加速现象。理论结论经大量数值实验验证，我们认为这一新视角将推动神经网络优化研究的深入。
原文摘要 · Abstract (English)
Even for the gradient descent (GD) method applied to neural network training, understanding its optimization dynamics, including convergence rate, iterate trajectories, function value oscillations, and especially its implicit acceleration, remains a challenging problem. We analyze nonlinear models with the logistic loss and show that the steps of GD reduce to those of generalized perceptron algorithms (Rosenblatt, 1958), providing a new perspective on the dynamics. This reduction yields significantly simpler algorithmic steps, which we analyze using classical linear algebra tools. Using these tools, we demonstrate on a minimalistic example that the nonlinearity in a two-layer model can provably yield a faster iteration complexity $\tilde{O}(\sqrt{d})$ compared to $Ω(d)$ achieved by linear models, where $d$ is the number of features. This helps explain the optimization dynamics and the implicit acceleration phenomenon observed in neural networks. The theoretical results are supported by extensive numerical experiments. We believe that this alternative view will further advance research on the optimization of neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。