arXiv:2605.11838cs.LGmath.OC2026-05

针对梯度矩阵结构提出谱裁剪,提升训练稳定性。

Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters

论文配图:Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
图 1 · 摘自论文原文
  • 按奇异值而非向量范数裁剪梯度,保留方向信息
  • 在重尾噪声下实现最优收敛速率 $\mathcal{O}(K^{\frac{2-2\alpha}{3\alpha-2}})$
  • 支持层内自适应阈值,适合大模型高效训练

梯度裁剪是应对噪声、重尾随机梯度训练神经网络的标准手段;然而,现有方法将所有参数视为向量,忽视了现代架构中参数的矩阵结构。我们实证发现,数据异常值通常仅放大层间梯度矩阵的少数几个主导奇异值,其余谱成分基本不变。受此启发,我们提出谱裁剪:对超过阈值的奇异值进行截断,同时保留奇异方向。该框架推广了经典梯度范数裁剪,可无缝集成至现有优化器。我们为带谱裁剪的SGD提供了非凸优化的收敛分析,得到重尾噪声下的最优收敛率 $\mathcal{O}\left(K^{\frac{2 - 2\alpha}{3\alpha - 2}}\right)$。为减少超参调优,我们引入基于移动平均或滑动窗口分位数的层内自适应阈值。最后,我们设计高效实现,仅通过随机截断SVD裁剪前 $r$ 个奇异值,避免大型层的完整分解。实验表明,在合成重尾场景与真实神经网络训练任务中均表现良好。

原文摘要 · Abstract (English)

Gradient clipping is a standard safeguard for training neural networks under noisy, heavy-tailed stochastic gradients; yet, most clipping rules treat all parameters as vectors and ignore the matrix structure of modern architectures. We show empirically that data outliers often amplify only a small number of leading singular values in layer-wise gradient matrices, while the rest of the spectrum remains largely unchanged. Motivated by this phenomenon, we propose spectral clipping, which stabilizes training by clamping singular values that exceed a threshold while preserving the singular directions. This framework generalizes classical gradient norm clipping and can be easily integrated into existing optimizers. We provide a convergence analysis for non-convex optimization with spectrally clipped SGD, yielding the optimal $\mathcal{O}\left(K^{\frac{2 - 2α}{3α- 2}}\right)$ rate for heavy-tailed noise. To minimize hyperparameter tuning, we introduce layer-wise adaptive thresholds based on moving averages or sliding-window quantiles of the top singular values. Finally, we develop efficient implementations that clip only the top $r$ singular values via randomized truncated SVD, avoiding full decompositions for large layers. We demonstrate competitive performance across synthetic heavy-tailed settings and neural network training tasks.

梯度裁剪谱方法优化器矩阵参数

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。