用可学习先验提升神经网络不确定性估计,性能媲美主流方法。
Last Layer Empirical Bayes
- 在最后一层引入可学习先验的归一化流,实现经验贝叶斯建模。
- 在多个数据集上达到与贝叶斯神经网络和深度集成相当的不确定性量化效果。
- 适合关注模型置信度评估与可靠性提升的研究者。
量化神经网络预测中的固有不确定性是人工智能的关键挑战。贝叶斯神经网络(BNNs)和深度集成是解决该问题的主要方法,二者均通过在权重分布上取期望来生成预测;前者使用后验分布,后者使用点质量混合分布。受近期研究启发——集成所用分布可视为对应于学习到的数据依赖先验的后验,本文提出最后层经验贝叶斯(LLEB)。LLEB将可学习先验建模为归一化流,并通过最大化证据下界进行训练;为保持可计算性,仅在最后一层使用该流。我们证明了LLEB的合理性,并展示了其在先验强度上介于标准BNN与集成之间。实验表明,LLEB性能与现有方法相当,凸显经验贝叶斯在不确定性量化研究中的潜力。
原文摘要 · Abstract (English)
The task of quantifying the inherent uncertainty associated with neural network predictions is a key challenge in artificial intelligence. Bayesian neural networks (BNNs) and deep ensembles are among the most prominent approaches to tackle this task. Both approaches produce predictions by computing an expectation of neural network outputs over some distribution on the corresponding weights; this distribution is given by the posterior in the case of BNNs, and by a mixture of point masses for ensembles. Inspired by recent work showing that the distribution used by ensembles can be understood as a posterior corresponding to a learned data-dependent prior, we propose last layer empirical Bayes (LLEB). LLEB instantiates a learnable prior as a normalizing flow, which is then trained to maximize the evidence lower bound; to retain tractability we use the flow only on the last layer. We show why LLEB is well motivated, and how it interpolates between standard BNNs and ensembles in terms of the strength of the prior that they use. LLEB performs on par with existing approaches, highlighting that empirical Bayes is a promising direction for future research in uncertainty quantification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。