通过无限宽网络推导出有限宽网络的不确定性，实现无需大量训练即可估算模型置信度。
Uncertainty Quantification From Scaling Laws in Deep Neural Networks
- 基于神经正切核初始化，研究无限宽与有限宽多层感知机的测试损失分布。
- 发现测试损失的变异系数在大数据量下趋于稳定，不随数据量变化。
- 适用于需要可靠不确定估计的物理科学领域，尤其适合小样本场景。
在物理科学中，量化机器学习结果的不确定性至关重要。本文聚焦于神经网络初始化分布带来的不确定性。我们计算了在无限宽极限下，采用神经正切核（NTK）初始化的多层感知机（MLP）测试损失 $\mathcal{L}$ 的均值 $μ_{\mathcal{L}}$ 和方差 $σ_{\mathcal{L}}^2$，并将其与三种任务（MNIST分类、CIFAR分类、量能器能量回归）的有限宽网络进行实证对比。观察到测试损失均值和方差均随训练集大小 $N_\mathcal{D}$ 呈幂律缩放；但关键发现是，变异系数 $ε_{\mathcal{L}} \equiv σ_{\mathcal{L}}/μ_{\mathcal{L}}$ 在足够大的 $N_\mathcal{D}$ 下，无论在无限宽还是有限宽情况下均趋于恒定。这意味着有限宽网络的变异系数可近似为其无限宽对应值，且理论上可通过有限宽微扰理论计算。
原文摘要 · Abstract (English)
Quantifying the uncertainty from machine learning analyses is critical to their use in the physical sciences. In this work we focus on uncertainty inherited from the initialization distribution of neural networks. We compute the mean $μ_{\mathcal{L}}$ and variance $σ_{\mathcal{L}}^2$ of the test loss $\mathcal{L}$ for an ensemble of multi-layer perceptrons (MLPs) with neural tangent kernel (NTK) initialization in the infinite-width limit, and compare empirically to the results from finite-width networks for three example tasks: MNIST classification, CIFAR classification and calorimeter energy regression. We observe scaling laws as a function of training set size $N_\mathcal{D}$ for both $μ_{\mathcal{L}}$ and $σ_{\mathcal{L}}$, but find that the coefficient of variation $ε_{\mathcal{L}} \equiv σ_{\mathcal{L}}/μ_{\mathcal{L}}$ becomes independent of $N_\mathcal{D}$ at both infinite and finite width for sufficiently large $N_\mathcal{D}$. This implies that the coefficient of variation of a finite-width network may be approximated by its infinite-width value, and may in principle be calculable using finite-width perturbation theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。