用单次前向传播实现贝叶斯深度学习高效预测
Streamlining Prediction in Bayesian Deep Learning
- 通过激活函数局部线性化与线性层局部高斯近似，避免采样
- 在回归和分类任务中达到与蒙特卡洛相当的精度
- 适用于MLP、ViT、GPT-2等模型，适合追求推理速度的场景
贝叶斯深度学习（BDL）近年来备受关注，但其推断（如预测）的高效计算长期被忽视，蒙特卡洛积分仍为标准方法。本文提出一种通过单次前向传播实现预测简化的方案：对激活函数进行局部线性化，在线性层采用局部高斯近似，从而解析计算后验预测分布的近似值。该方法在MLP、ViT和GPT-2等模型上均验证有效，涵盖回归与分类任务。实验表明，其性能接近蒙特卡洛方法，同时显著提升推理效率。开源代码见：https://github.com/AaltoML/SUQ。
原文摘要 · Abstract (English)
The rising interest in Bayesian deep learning (BDL) has led to a plethora of methods for estimating the posterior distribution. However, efficient computation of inferences, such as predictions, has been largely overlooked with Monte Carlo integration remaining the standard. In this work we examine streamlining prediction in BDL through a single forward pass without sampling. For this we use local linearisation on activation functions and local Gaussian approximations at linear layers. Thus allowing us to analytically compute an approximation to the posterior predictive distribution. We showcase our approach for both MLP and transformers, such as ViT and GPT-2, and assess its performance on regression and classification tasks. Open-source library: https://github.com/AaltoML/SUQ
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。