解析神经网络层间曲率，揭示梯度优化背后的结构机制。
Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures
- 将损失函数的海森矩阵按图结构分解，分离出凸部分与残余曲率。
- 发现ReLU网络输入层的残余曲率项为零，参数层仍存非吉布斯-牛顿项。
- 提出共振、耦合等诊断指标，可快速评估层间曲率交互，适合模型调优者。
现代自动微分框架（如JAX、PyTorch）返回的损失函数海森矩阵是单一张量，未暴露层间相互作用的内部结构。本文提出一种分析形式，显式将任意架构的海森矩阵按有向无环图（DAG）分解为块。标准分解 $H = H^{GN} + H^T$ 将高斯-牛顿部分（凸项）与张量部分（导致鞍点的残余曲率）分离。对于分段线性激活函数（如ReLU），输入海森矩阵的张量部分几乎处处为零（$H^{T}_{v,w} equiv0$ a.e.），而 $H^f_{v,w}=H^{GN}_{v,w} succeq0$；完整参数海森矩阵包含不归约为吉布斯-牛顿形式的残余项。基于此分解，我们引入诊断指标（层间共振 $/mathcal{R}$、几何耦合 $/mathcal{C}$、稳定秩 $/mathcal{D}$、GN-Gap），可在 $O(P)$ 时间内随机估计，揭示层间曲率交互结构。理论分析解释了普通网络中共振的指数衰减及跳跃连接下的保持现象。实证验证涵盖全连接MLP（实验1–5）和卷积结构（ResNet-18，约1100万参数，实验6）。当架构退化为单节点时，所有定义退化为标准海森矩阵 $ abla^2_θ/mathcal{L}(θ) obR^{p imes p}$。
原文摘要 · Abstract (English)
Modern automatic differentiation frameworks (JAX, PyTorch) return the Hessian of the loss function as a monolithic tensor, without exposing the internal structure of inter-layer interactions. This paper presents an analytical formalism that explicitly decomposes the full Hessian into blocks indexed by the DAG of an arbitrary architecture. The canonical decomposition $H = H^{GN} + H^T$ separates the Gauss--Newton component (convex part) from the tensor component (residual curvature responsible for saddle points). For piecewise-linear activations (ReLU), the tensor component of the input Hessian vanishes ($H^{T}_{v,w}\!\equiv\!0$ a.e., $H^f_{v,w}\!=\!H^{GN}_{v,w}\!\succeq\!0$); the full parametric Hessian contains residual terms that do not reduce to the GGN. Building on this decomposition, we introduce diagnostic metrics (inter-layer resonance~$\mathcal{R}$, geometric coupling~$\mathcal{C}$, stable rank~$\mathcal{D}$, GN-Gap) that are estimated stochastically in $O(P)$ time and reveal structural curvature interactions between layers. The theoretical analysis explains exponential decay of resonance in vanilla networks and its preservation under skip connections; empirical validation spans fully connected MLPs (Exp.\,1--5) and convolutional architectures (ResNet-18, ${\sim}11$M~parameters, Exp.\,6). When the architecture reduces to a single node, all definitions collapse to the standard Hessian $\nabla^2_θ\mathcal{L}(θ)\in\mathbb{R}^{p\times p}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。