用低秩方法提升贝叶斯神经网络的不确定性估计效率与精度。
Low Rank Based Subspace Inference for the Laplace Approximation of Bayesian Neural Networks
- 基于低秩技术构建最优子空间模型，实现高效贝叶斯推断。
- 降维后的协方差矩阵近似结果接近全量级拉普拉斯近似。
- 提出可扩展方案并提供评估不同子空间模型的定量指标。
神经网络的子空间推断假设参数空间的低维子集即可实现可靠的不确定性量化。本文通过低秩技术验证该假设的有效性，推导出基于拉普拉斯近似的子空间模型表达式，其在特定数据集下具有理论最优性。实验表明，采用降维协方差矩阵构建的拉普拉斯近似能紧密逼近使用精确协方差矩阵的完整近似。当可行时，该子空间模型可作为基准用于评估其他子空间模型性能。此外，本文提出一种实用且可扩展的近似方法，并与文献中现有方法对比，结果表明本方法整体表现更优。同时，我们设计了一种无需精确拉普拉斯近似即可定性比较不同子空间模型质量的度量指标。
原文摘要 · Abstract (English)
Subspace inference for neural networks assumes that a subspace of their parameter space suffices to produce a reliable uncertainty quantification. In this work, we underpin the validity of this assumption by using low rank techniques. We derive an expression for a subspace model to a Bayesian inference scenario based on the Laplace approximation that is, in a certain sense, optimal given a specific dataset. We empirically show that a Laplace approximation constructed with a dimensionally reduced covariance matrix closely matches the full Laplace approximation obtained using the exact covariance matrix. Where feasible, this subspace model can serve as a baseline for benchmarking the performance of subspace models. In addition, we provide a scalable approximation of this subspace construction that is usable in practice and compare it to existing subspace models from the literature. In general, our approximation scheme outperforms previous work. Furthermore, we present a metric to qualitatively compare the approximation quality of different subspace models even if the exact Laplace approximation is unknown.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。