MFVI在预测方差上可能高估，而非仅低估。
Gaussian Mean Field Variational Inference can Overestimate Predictive Variance
- 通过共轭贝叶斯线性回归分析发现MFVI预测方差会高估。
- 在训练数据集中，其预测方差反而高于精确后验。
- 适合关注变分推断可靠性与冷后验效应的研究者。
均值场变分推断（MFVI）通常被认为低估后验方差。通过对共轭贝叶斯线性回归（BLR）的分析，我们发现这一认识不完整：尽管MFVI在参数空间中低估了方差，但在预测空间中却可能高估预测方差。我们证明，若MFVI在某些方向上低估预测方差，则必然在其他方向上高估。关键在于，这种高估发生在训练数据集聚集的方向上。因此，当测试点来自训练分布时，MFVI的期望预测方差超过了精确后验。我们展示了一个病态案例：在分布内数据上，MFVI的预测方差甚至未低于先验。我们将此现象与冷后验效应联系起来，指出调整温度可纠正该高估，使预测更接近精确后验。理论在合成与真实回归任务中得到验证。
原文摘要 · Abstract (English)
Mean Field Variational Inference (MFVI) is widely understood to underestimate posterior variance. By analysing conjugate Bayesian Linear Regression (BLR), we show that this characterization is incomplete: while MFVI underestimates the variance in parameter space, it can overestimate the predictive variance compared to the exact posterior. We show that if the MFVI posterior underestimates predictive variances in some directions, it necessarily overestimates them in others. Crucially, this overestimation occurs in directions where the training data concentrates. This leads to the surprising result that, for a test point drawn from the training distribution, MFVI's expected predictive variance exceeds that of the exact posterior. We demonstrate a pathological case of this effect, where the MFVI posterior fails to reduce predictive variance compared to the prior on in distribution data. We connect these results to the Cold Posterior Effect, arguing that varying the temperature can correct this overestimation, yielding predictions closer to those of the exact posterior. We validate our theory on synthetic and real-world regression tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。