spectral gradient 有效解决相位恢复中的方向错位问题,提升收敛稳定性。
Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval
- 用谱梯度更新保留方向信息、去除尺度影响,避免高方差方向干扰
- 在强各向异性下,标准梯度下降会放大无关方向,导致信号对齐失败
- 适合研究优化算法机制或相位恢复任务的科研人员参考
谱梯度方法(如Muon优化器)通过保留梯度方向而舍弃其尺度,在深度学习中表现出优异的实证性能。本文通过非线性相位恢复模型的动力学分析,揭示其优势机制:在具有各向异性高斯输入的两层神经网络(二次激活、固定第二层权重)中,当主方差方向与信号正交时,标准梯度下降(GD)在早期逃逸阶段会因高方差但无信息的尖峰方向被乘法放大,导致与真实信号的对齐严重退化。而谱梯度下降(SpecGD)能消除该尖峰放大效应,实现稳定的信号对齐并加速噪声收缩。数值实验验证了理论结果,且这些现象在更广泛的各向异性协方差下依然存在。
原文摘要 · Abstract (English)
Spectral gradient methods, such as the Muon optimizer, modify gradient updates by preserving directional information while discarding scale, and have shown strong empirical performance in deep learning. We investigate the mechanisms underlying these gains through a dynamical analysis of a nonlinear phase retrieval model with anisotropic Gaussian inputs, equivalent to training a two-layer neural network with the quadratic activation and fixed second-layer weights. Focusing on a spiked covariance setting where the dominant variance direction is orthogonal to the signal, we show that gradient descent (GD) suffers from a variance-induced misalignment: during the early escaping stage, the high-variance but uninformative spike direction is multiplicatively amplified, degrading alignment with the true signal under strong anisotropy. In contrast, spectral gradient descent (SpecGD) removes this spike amplification effect, leading to stable alignment and accelerated noise contraction. Numerical experiments confirm the theory and show that these phenomena persist under broader anisotropic covariances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。