arXiv:2604.14870cs.LG2026-04

针对神经网络损失曲面的局部稳定性,提出基于曲率对齐的高效探测方法。

Curvature-Aligned Probing for Local Loss-Landscape Stabilization

论文配图:Curvature-Aligned Probing for Local Loss-Landscape Stabilization
图 1 · 摘自论文原文
  • 基于经验海森矩阵的前D个主成分方向探测损失变化,聚焦关键变形方向。
  • 仅用局部二次模型即实现与全参数空间相当的稳定率,且计算复杂度由子空间维数决定。
  • 新方法在小参数占比下逼近全空间结果,且闭式估计速度比蒙特卡洛快多个数量级。

在样本增长下,局部损失曲面的稳定性通常通过参数空间中的点对点或各向同性平均来衡量。然而,这两种方法探测的方向对强各向异性的神经网络损失曲面主导变形贡献甚微。本文将稳定性评估重构为观测问题,引入一个由聚合阶数和探测分布参数化的统一准则族;在此族中提出曲率对齐准则 $Δ_2^{(D)}$,该准则在训练解附近的经验海森矩阵的前-D-个特征空间内探测损失增量场。仅基于局部二次模型,我们证明 $Δ_2^{(D)}$ 保持了全空间准则的 $O(k^{-2})$ 均方收敛速率,同时将环境维度的曲率依赖替换为子空间维度 $D$ 的依赖;推论给出闭式谱表达式,命题则表明前-D-个特征空间在特征空间对齐族中为极值。我们还推导出基于海森-向量积、子空间蒙特卡洛以及闭式高斯矩代理的可扩展估计器。在一个仅解码器的Transformer上,占据极小参数空间比例的曲率对齐探测器已能在验证的局部范围内以数值噪声水平复现全空间均方信号,且闭式估计器在子空间构建后比直接蒙特卡洛快多个数量级。

原文摘要 · Abstract (English)

Local loss-landscape stabilization under sample growth is typically measured either pointwise or through isotropic averaging in the full parameter space. Despite practical value, both choices probe directions that contribute little to the dominant local deformation of strongly anisotropic neural landscapes. We recast stabilization as an observational problem and introduce a unified family of criteria parameterized by an aggregation order and a probing distribution; within this family we propose a curvature-aligned criterion $Δ_2^{(D)}$ that probes the loss increment field in the top-$D$ eigenspace of the empirical Hessian near a trained solution. Solely from a local quadratic model, we prove that $Δ_2^{(D)}$ preserves the $O(k^{-2})$ mean-squared rate of the full-space criterion while replacing ambient-dimension curvature dependence with dependence on the subspace dimension $D$; a corollary gives a closed-form spectral expression and a proposition identifies the top-$D$ eigenspace as extremal within the eigenspace-aligned family. We also derive scalable estimators based on Hessian-vector products, subspace Monte Carlo, and a closed-form Gaussian-moment proxy. On a decoder-only transformer, a curvature-aligned probe occupying a tiny fraction of parameter space already reproduces the full-space mean-squared signal to within numerical noise throughout the validated local regime, and the closed-form estimator is orders of magnitude faster than direct Monte Carlo after subspace construction.

损失曲面曲率对齐高效探测海森矩阵

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。