用粒子混合模型改进贝叶斯推断，解决方差坍缩问题。
ELBOing Stein: Variational Bayes with Stein Mixture Inference
- 每个粒子构建混合分布的成分，提升不确定性建模能力。
- 在小规模贝叶斯神经网络上，用更少粒子达到更高精度。
- 适合处理高维宽数据的变分贝叶斯推断，可定制引导分布。
Stein变分梯度下降（SVGD）通过一组粒子近似后验分布，但存在方差坍缩问题，即使在中小规模贝叶斯神经网络（BNNs）中也会低估不确定性。为此，本文提出斯坦因混合推断（SMI），让每个粒子参数化混合模型的一个成分，并优化证据下界（ELBO），引入用户指定的由粒子参数化的引导分布。SMI扩展了非线性SVGD框架至变分贝叶斯场景，在标准数据集上有效避免了方差坍缩，且在小规模BNN中用更少粒子即可准确估计不确定性。结合非线性SVGD、ELBO优化与可定制引导，SMI为高维宽数据的变分贝叶斯推断提供了有前景的解决方案。
原文摘要 · Abstract (English)
Stein variational gradient descent (SVGD) [Liu and Wang, 2016] performs approximate Bayesian inference by representing the posterior with a set of particles. However, SVGD suffers from variance collapse, i.e. poor predictions due to underestimating uncertainty [Ba et al., 2021], even for moderately-dimensional models such as small Bayesian neural networks (BNNs). To address this issue, we generalize SVGD by letting each particle parameterize a component distribution in a mixture model. Our method, Stein Mixture Inference (SMI), optimizes a lower bound to the evidence (ELBO) and introduces user-specified guides parameterized by particles. SMI extends the Nonlinear SVGD framework [Wang and Liu, 2019] to the case of variational Bayes. SMI effectively avoids variance collapse, judging by a previously described test developed for this purpose, and performs well on standard data sets. In addition, SMI requires considerably fewer particles than SVGD to accurately estimate uncertainty for small BNNs. The synergistic combination of NSVGD, ELBO optimization and user-specified guides establishes a promising approach towards variational Bayesian inference in the case of tall and wide data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。