用固定均值高斯过程提升预训练模型的不确定性估计
Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning
- 将预训练网络输出作为高斯过程均值，仅优化方差以估计置信度
- 在ImageNet上训练成本与数据量无关，效率远超现有方法
- 无需修改模型结构，适用于任意深度网络架构
近年来，针对预训练深度神经网络（DNN）进行后验不确定性估计的研究日益增多。在通过反向传播训练好DNN后，这些方法通过添加输出置信度度量（如误差条）来增强原始网络，同时保持初始精度。本文提出一种新型稀疏变分高斯过程（GPs）家族，当使用通用核函数时，其后验均值可固定为任意连续函数。具体地，我们将该GP的均值固定为预训练DNN的输出，使方法能有效拟合预测方差以估计DNN的预测不确定性。该方法采用变分推断（VI）实现高效随机优化，训练开销与训练样本数量无关，可高效扩展至ImageNet等大规模数据集。所提方法称为固定均值高斯过程（FMGP），具有架构无关性，仅依赖预训练模型输出来调整预测方差。实验表明，相比当前最优的DNN后验贝叶斯推理方法，FMGP在不确定性估计和计算效率方面均有显著提升。
原文摘要 · Abstract (English)
Recently, there has been an increasing interest in performing post-hoc uncertainty estimation about the predictions of pre-trained deep neural networks (DNNs). Given a pre-trained DNN via back-propagation, these methods enhance the original network by adding output confidence measures, such as error bars, without compromising its initial accuracy. In this context, we introduce a novel family of sparse variational Gaussian processes (GPs), where the posterior mean is fixed to any continuous function when using a universal kernel. Specifically, we fix the mean of this GP to the output of the pre-trained DNN, allowing our approach to effectively fit the GP's predictive variances to estimate the DNN prediction uncertainty. Our approach leverages variational inference (VI) for efficient stochastic optimization, with training costs that remain independent of the number of training points, scaling efficiently to large datasets such as ImageNet. The proposed method, called fixed-mean GP (FMGP), is architecture-agnostic, relying solely on the pre-trained model's outputs to adjust the predictive variances. Experimental results demonstrate that FMGP improves both uncertainty estimation and computational efficiency when compared to state-of-the-art methods for DNN post-hoc Bayesian inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。