通过温度调控改进贝叶斯神经网络的不确定性估计,提升泛化性能。
On Cold Posteriors of Probabilistic Neural Networks: Understanding the Cold Posterior Effect and A New Way to Learn Cold Posteriors with Tight Generalization Guarantees
- 引入温度调节机制优化后验分布,实现更精准的不确定性建模。
- 理论证明冷后验在 PAC-Bayes 框架下具备严格泛化保证。
- 适用于需要高可信度预测的场景,如医疗或自动驾驶。
贝叶斯推断通过贝叶斯定理结合先验知识与观测数据,为不确定性提供严谨的概率框架。在贝叶斯深度学习中,神经网络权重被视为具有先验分布的随机变量,从而实现对预测不确定性的概率解释。然而,现有贝叶斯方法缺乏对未见数据的理论泛化保证。PAC-Bayesian 分析通过频率学框架为随机预测器提供泛化界,从而验证贝叶斯方法在机器学习中的可靠性。温度 $T$(或逆温度 $λ=1/T$)源自统计物理,在贝叶斯推断与 PAC-Bayesian 分析中自然出现。当 $T<1$(冷后验),似然项被加权放大,后验分布更尖锐;当 $T>1$(热后验),似然项被削弱,后验更弥散。通过调节数据与先验正则化的平衡,温度调整可缓解模型欠拟合或过拟合问题,提升预测性能。
原文摘要 · Abstract (English)
Bayesian inference provides a principled probabilistic framework for quantifying uncertainty by updating beliefs based on prior knowledge and observed data through Bayes' theorem. In Bayesian deep learning, neural network weights are treated as random variables with prior distributions, allowing for a probabilistic interpretation and quantification of predictive uncertainty. However, Bayesian methods lack theoretical generalization guarantees for unseen data. PAC-Bayesian analysis addresses this limitation by offering a frequentist framework to derive generalization bounds for randomized predictors, thereby certifying the reliability of Bayesian methods in machine learning. Temperature $T$, or inverse-temperature $λ= \frac{1}{T}$, originally from statistical mechanics in physics, naturally arises in various areas of statistical inference, including Bayesian inference and PAC-Bayesian analysis. In Bayesian inference, when $T < 1$ (``cold'' posteriors), the likelihood is up-weighted, resulting in a sharper posterior distribution. Conversely, when $T > 1$ (``warm'' posteriors), the likelihood is down-weighted, leading to a more diffuse posterior distribution. By balancing the influence of observed data and prior regularization, temperature adjustments can address issues of underfitting or overfitting in Bayesian models, bringing improved predictive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。