提出精确高斯矩匹配方法，显著提升残差网络的不确定性传播精度。
Exact Gaussian Moment Matching for Residual Networks: a Second-Order Method
- 基于逐层矩匹配，实现多种激活函数的精确均值与协方差传播。
- 在随机网络上，KL散度误差降低百万倍；在变分贝叶斯模型中提升百倍。
- 理论证明可消除低方差主导误差，适合需要精准不确定性的场景。
我们研究将一般多元高斯分布的均值与协方差通过深层（残差）神经网络进行逐层传播的问题，采用层间矩匹配方法。本文首次对probit、GeLU、ReLU（作为GeLU极限）、Heaviside（作为probit极限）及正弦激活函数实现了精确的矩匹配，涵盖前馈与广义残差层。在随机网络中，我们的方法在KL散度误差指标上相较主流替代方案提升数个数量级，最高达百万倍；在变分贝叶斯神经网络中，相比最先进的确定性推断方法，其从蒙特卡洛真值出发的KL散度降低近百倍。此外，我们还给出了平滑距离误差界，表明在正则性假设下，该方法可消除主导的低方差误差，并将高阶局部精度沿网络层级传递。
原文摘要 · Abstract (English)
We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On random networks, we find orders-of-magnitude improvements in the KL divergence error metric, up to a millionfold, over popular alternatives. On a variational Bayes neural network, we show that our method attains hundredfold improvements in KL divergence from Monte Carlo ground truth over a state-of-the-art deterministic inference method. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。