用高阶展开分析有限宽度神经网络输出的分布偏差
Optimal Non-Asymptotic Edgeworth Expansions for Multivariate Neural Network Outputs
- 提出任意阶数的多维埃德沃斯展开，刻画有限宽网络输出偏离高斯极限的规律
- 证明总变差距离收敛速率达 n^{-m}，且该速率紧致，不可改进
- 适用于神经网络后验分布误差量化，也推广至条件高斯序列收敛场景
具有高斯初始化权重的有限宽度全连接神经网络在有限输入点上的输出会偏离其无限宽度下的高斯极限，表现出不消失的高阶累积量。我们针对这类输出，采用任意阶数为 $4m-1$ 的多维埃德沃斯展开进行逼近，其中 $m\in\mathbb{N}$。在假设对应高斯极限具有可逆协方差矩阵、激活函数多项式有界的情况下，建立了真实网络输出分布与埃德沃斯近似之间总变差距离的上界，阶数为 $n^{-m}$，并给出了匹配的下界。作为应用，我们量化了当先验被其埃德沃斯展开替代时贝叶斯后验分布的误差。所得结果更一般，亦适用于一系列条件高斯向量收敛到具有可逆协方差的高斯向量的情形。
原文摘要 · Abstract (English)
Finite-width fully connected neural networks with Gaussian-initialized weights deviate from their infinite-width Gaussian limit, exhibiting non-vanishing higher-order cumulants. We approximate these deviations, for a neural network evaluated in a finite number of inputs, using multidimensional Edgeworth expansions of arbitrary order $4m-1$, with $m\in\mathbb{N}$. Assuming that the corresponding Gaussian limit has an invertible covariance matrix and that the activation function is polynomially bounded, we establish a bound of order $n^{-m}$ on the total variation distance between the law of the true network output and its Edgeworth approximation, with matching lower bounds. As an application, we quantify the error in Bayesian posterior distributions when the prior is replaced by its Edgeworth expansion. Our results are more general and also apply to sequences of conditionally Gaussian vectors converging to a Gaussian vector with invertible covariance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。