arXiv:2606.02596cs.LG2026-06

揭示神经网络损失曲率指数差异的几何根源,提出可验证的谱分解模型。

Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

  • 通过谱对齐分解公式,将曲率指数变化归因于梯度方向与特征基的几何对齐。
  • 实证发现93个层中曲率指数与梯度秩衰减率的乘积误差仅2%,无自由参数。
  • 适用于理解各层结构特性,适合研究优化器设计与模型几何的读者。

神经网络损失曲面的曲率指数α在不同层类型中系统性变化(卷积层α≈2,Transformer注意力层α≈1,MLP上投影层α<1)。我们证明了谱对齐分解公式:α = 2 + d logΦ_k / d logσ_k,其中Φ_k衡量Kronecker因子特征基与梯度奇异方向之间的对齐程度。该分解将‘为何α变化’转化为几何问题,并对LayerNorm、残差连接和softmax头给出了答案。该式导出谱传递恒等式 s = αγ,关联曲率指数α、有效梯度秩衰减率γ与海森矩阵衰减指数s。该恒等式为代数关系,其经验内容是:在93个层、五种架构、三个数据集上,α与γ分别通过HVP和SVD独立拟合,恢复的s值中位误差仅约2%。泽塔函数界表明每层曲率集中在单一有效方向。作为概念验证,我们推导出自适应预条件器T(σ;α),并在视觉任务中显示基于T的谱牛顿法优于AdamW(当α≈2时)。

原文摘要 · Abstract (English)

The curvature exponent $α$ in $h_k \propto σ_k^α$ -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types ($α\approx 2$ for convolutions, $\approx 1$ for transformer attention, $< 1$ for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: $α= 2 + d\logΦ_k / d\logσ_k$, where $Φ_k$ measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does $α$ vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity $s = αγ$ linking curvature exponent, effective gradient rank-decay $γ$, and Hessian decay exponent $s$. The identity is algebraic; its empirical content is that $α$ and $γ$, fit on independent data (HVPs vs. SVD), recover $s$ to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner $T(σ;α)$ and show that Spectral Newton -- implementing $T$ in the gradient singular basis -- outperforms AdamW on vision benchmarks where $α\approx 2$.

曲率分析优化器设计神经网络几何谱分解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。