提出高效计算神经网络度量张量的确定性界与无偏随机估计方法。
Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds
- 从概率分布核心空间出发，分析其费雪信息矩阵谱与包络。
- 推导出神经网络流形上度量张量的确定性上下界。
- 基于Hutchinson方法实现单次反向传播即可完成的无偏估计。
深度神经网络的高维参数空间——神经流形——具有由费雪信息定义的独特度量张量。可靠且可扩展地计算该张量对理论与实践均具价值。针对神经分类器，本文回归到概率分布的低维核心空间，研究其费雪信息矩阵的谱与包络，并将发现推广至神经流形上度量张量的确定性边界。提出一种基于Hutchinson迹估计的无偏随机估计器，可每批仅需一次反向传播高效评估，其标准差在缩放意义下不超过真实值。
原文摘要 · Abstract (English)
The high-dimensional parameter space of deep neural networks -- the neuromanifold -- is endowed with a unique metric tensor defined by the Fisher information. Reliable and scalable computation of this metric tensor is valuable for theorists and practitioners. Focusing on neural classifiers, we return to a low-dimensional space of probability distributions, which we call the core space, and examine the spectrum and envelopes of its Fisher information matrix. We extend our discoveries there to deterministic bounds for the metric tensor on the neuromanifold. We introduce an unbiased random estimator based on Hutchinson's trace method and derive related bounds. It can be evaluated efficiently with a single backward pass per batch, with a standard deviation bounded by the true value up to scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。