提出一种高效估算神经网络每层海森迹的随机方法,可用于训练过程监控。
Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training
- 用单次反向传播结合随机估计,快速计算各层海森迹。
- 在真实数据集上实现接近100%的错误标签记忆检测率。
- 适合关注训练稳定性与模型行为分析的研究者使用。
损失值和梯度范数对健康与异常训练状态的区分能力较弱,而经验风险的曲率在两类状态间存在质的区别,但在参数量达 $10^6$ 至 $10^8$ 时难以直接获取。本文提出一种针对神经网络经验风险海森矩阵对角块迹的随机估计方法,结合Hutchinson迹估计与全参数向量的单次海森-向量乘积,在一次反向传播中无偏估计每层迹。研究发现权重共享下需在二次微分前组装层间海森矩阵;若将共享权重展开为独立变量,会引入系统偏差,其符号与大小由展开后海森矩阵的跨实例块决定。推导出固定海森下的估计器方差闭式表达,并分解了小批量采样下的总方差,得到关键探针数 $K^{ar{}}$,支持在线监控中 $K\in[5,10]$ 的实用建议。该方法应用于ResNet-18、ResNet-34和VGG-11在CIFAR-10与CIFAR-100上的标签记忆检测,基于校准的累积和决策规则,达到179/180的检测成功率,误报率为16/120。
原文摘要 · Abstract (English)
The loss and the norm of its gradient separate the healthy and the pathological regimes of neural-network training only weakly, whilst the curvature of the empirical risk differs qualitatively between them but is inaccessible explicitly at parameter counts $P\sim 10^{6}-10^{8}$. We present a stochastic estimator of the trace of the diagonal blocks of the Hessian matrix of the empirical risk of a neural network. The procedure combines the Hutchinson stochastic trace estimator with a single Hessian-vector product over the whole parameter vector and recovers unbiased estimates of every per-layer trace in one backward pass through the computational graph. We show that correctness under weight sharing requires the layer-wise Hessian to be assembled before the second differentiation: unrolling shared weights into independent coordinates introduces a systematic bias whose sign and magnitude are governed by the cross-instance blocks of the unrolled Hessian. A closed-form expression for the variance of the estimator at a fixed Hessian is derived, together with a decomposition of the total variance under the mini-batch sampling distribution. This decomposition yields a critical probe count $K^{\star}$ that balances the two sources of randomness and supports the practical recommendation $K\in[5,10]$ in the on-line monitoring regime. The estimator is applied to the detection of the label-memorisation regime of ResNet-18, ResNet-34, and VGG-11 on CIFAR-10 and CIFAR-100, where a calibrated cumulative-sum decision rule attains an empirical detection power of $179/180$ at a false-alarm rate of $16/120$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。