揭示耦合梯度下降中瞬时放大现象的理论边界，指导高维学习系统设计。
Pseudospectral Bounds for Transient Amplification in Coupled Gradient Descent
- 基于块三角雅可比矩阵构建紧致伪谱理论，量化非正规性导致的瞬时放大。
- 给出有限步迭代复杂度上界为 O(K(J)^2 log(1/δ))，适用于高维学习场景。
- 适用于双时间尺度优化、生成对抗网络等系统，尤其关注非渐近性能分析。
耦合梯度下降广泛存在于双层优化、双时间尺度随机逼近及生成对抗网络中。当耦合雅可比矩阵为块三角形时，渐近稳定性由对角块的谱半径决定，但收敛前的瞬时放大可能因非正规性而无限大。本文针对形如 J = [[A, 0], [C, D]] 的块三角雅可比矩阵，建立紧致伪谱理论，证明在 ρ(A), ρ(D) ≤ γ < 1 且 A、D 对称条件下，Kreiss 常数满足 K(J) ≤ 2/(1−γ) + ||C||/(4(1−γ))，并给出匹配的极小极大下界。刻画了谱不稳定的临界耦合阈值，通过诺伊曼级数扰动框架将理论推广至近自引用系统。由此获得有限时域迭代复杂度上界为 O(K(J)^2 log(1/δ))。以随机双时间尺度优化的尺度律形式呈现，揭示了传统谱半径分析无法捕捉的非渐近、实例相关高维学习动态。线性二次问题、IQC对比与神经网络训练实验验证了理论有效性。
原文摘要 · Abstract (English)
Coupled gradient descent - where the update of one parameter depends on another - arises naturally in bilevel optimization, two-time-scale stochastic approximation, and generative adversarial networks. When the coupled Jacobian is block-triangular, asymptotic stability is determined by the spectral radii of the diagonal blocks, yet transient amplification before convergence can be arbitrarily large due to non-normality. We develop a sharp pseudospectral theory for block-triangular Jacobians J = [[A, 0], [C, D]], proving Kreiss-constant bounds of the form K(J) <= 2/(1-γ) + ||C||/(4(1-γ)) when ρ(A), ρ(D) <= γ< 1 and A, D are symmetric, and establishing matching minimax lower bounds. We characterize the critical coupling threshold for spectral instability and extend the theory to nearly self-referential systems via a Neumann-series perturbation framework. As a consequence, we obtain a finite-horizon O(K(J)^2 log(1/δ)) iteration complexity bound. Framed as scaling laws for stochastic two-time-scale optimization, our results expose a non-asymptotic, instance-dependent regime of high-dimensional learning dynamics that is invisible to spectral-radius analysis. Experiments on linear-quadratic problems, IQC-based comparisons, and neural-network training confirm the theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。