揭示神经网络训练不稳定的非正规谱特征,提供早期预警指标。
Non-normal spectral signatures of instability in neural network training dynamics

- 用非正规算子理论分析优化器更新机制,发现其普遍非正规。
- κ(V)指标可提前预警瞬态放大,比谱半径更敏感,相差约一个数量级。
- 适用于研究自适应优化器稳定性,为诊断训练异常提供新视角。
深度网络训练中的不稳定性(如损失突增、振荡收敛、梯度异常)虽广泛存在,却缺乏严格的算子理论解释。本文表明,实际使用的优化器线性化更新算子通常是非正规的:Adam的非正规性由海森矩阵与对角自适应预条件器的换位子 [H, M] 控制,SGD with momentum 则源于更新映射的增广状态空间结构。将非正规稳定性理论应用于这些算子,我们推导出保守的伪谱前兆界,其中 κ(V) 作为瞬态放大的早期预警信号,即使谱半径低于1也能起作用;并证明更新算子的例外点是 κ(V) → ∞ 的极限情形。两层网络的数值实验表明,谱半径 ρ(J) 无法区分稳定与不稳定阶段,而 κ(V) 可将其分离开,差距约一个数量级。该结果补充了经典尖锐性准则,提供了连续的非正规放大严重程度度量。研究确立了非厄米算子理论在神经网络优化稳定性中的价值,为理解自适应优化稳定性提供了诊断语言和概念验证基准。
原文摘要 · Abstract (English)
Training instabilities in deep networks - loss spikes, oscillatory convergence, and gradient pathologies - are empirically prevalent but lack a rigorous operator-theoretic explanation. We show that the linearized update operators for practically used optimizers are generically non-normal: for Adam, non-normality is controlled by the commutator [H, M] between the Hessian and the diagonal adaptive preconditioner, while for SGD with momentum it arises from the augmented state-space structure of the update map. Applying non-normal stability theory to these operators, we derive a conservative pseudospectral precursor bound in which κ(V) serves as an early-warning indicator of transient amplification even when the spectral radius remains below one, and we establish that exceptional points of the update operator appear as the κ(V) -> \infty limiting case of this framework. Numerical experiments on two-layer networks confirm that the spectral radius ρ(J) provides no separation between stable and unstable training phases while κ(V) separates them by approximately one order of magnitude, complementing the classical sharpness criterion with a continuous severity measure of non-normal amplification. These results establish non-Hermitian operator theory as a useful and underexplored framework for neural network optimization stability, offering a diagnostic language and proof-of-concept benchmark for understanding adaptive optimization stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。