分析梯度下降中前向后向扰动的累积效应，解释深度学习训练中的梯度突变现象。
On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes
- 考虑前向后向双重扰动，揭示其在计算图中几何级放大机制。
- 证明非凸与普吕克-洛贾斯维茨条件下收敛性，给出不恶化渐近阶的条件。
- 解释训练中梯度突增为何发生或消失，适合研究优化稳定性者阅读。
我们研究带有 $N$ 个顺序算子的复合优化问题中，随机梯度下降（SGD）在前向与后向传递存在扰动的情况。不同于传统将梯度噪声视为加性且局部化的分析，中间输出与梯度的扰动会沿计算图传播并随算子数几何级放大。本文首次对该设定进行完整理论分析：刻画了前向与后向扰动在单步梯度更新中的传播与放大行为；推导了针对一般非凸目标函数及满足 Polyak--Łojasiewicz 条件函数的收敛保证；并识别出扰动不会降低渐近收敛阶的条件。作为副产品，本分析为深度学习中广泛观测到的梯度突增现象提供了理论解释，精确刻画了训练何时能从突增恢复或走向发散。在带凸与非凸正则化逻辑回归上的实验验证了理论预测，展示了预期的突增行为以及前向与后向扰动的不对称敏感性。
原文摘要 · Abstract (English)
We study stochastic gradient descent (SGD) for composite optimization problems with $N$ sequential operators subject to perturbations in both the forward and backward passes. Unlike classical analyses that treat gradient noise as additive and localized, perturbations to intermediate outputs and gradients cascade through the computational graph, compounding geometrically with the number of operators. We present the first comprehensive theoretical analysis of this setting. Specifically, we characterize how forward and backward perturbations propagate and amplify within a single gradient step, derive convergence guarantees for both general non-convex objectives and functions satisfying the Polyak--Łojasiewicz condition, and identify conditions under which perturbations do not deteriorate the asymptotic convergence order. As a byproduct, our analysis furnishes a theoretical explanation for the gradient spiking phenomenon widely observed in deep learning, precisely characterizing the conditions under which training recovers from spikes or diverges. Experiments on logistic regression with convex and non-convex regularization validate our theories, illustrating the predicted spike behavior and the asymmetric sensitivity to forward versus backward perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。