为变分推断提供基于稳定性的泛化界,可更准确评估模型性能。
Stability-based Generalization Bounds for Variational Inference
- 通过稳定性分析构建针对变分推断的算法特异性泛化界。
- 在现代神经网络上得到非平凡的泛化误差上界,效果优于传统方法。
- 适用于使用随机梯度下降优化的近似贝叶斯算法,适合研究者评估算法差异。
变分推断(VI)在贝叶斯机器学习中广泛用于近似推断。尽管已有基于PAC-Bayes分析的泛化界,但另一类通过稳定性或互信息约束的算法特异性界虽更紧致,却不直接适用于近似贝叶斯算法。本文填补了这一空白,为一类包含变分推断的近似贝叶斯算法(当使用随机梯度下降优化目标时)建立了基于稳定性的泛化界。与非贝叶斯情形类似,泛化误差由扰动数据集上的参数期望差异界定。新方法补充了PAC-Bayes分析,某些情况下能提供更紧的界。实验表明,该方法在现代神经网络架构和数据集上可获得非平凡的泛化界,并能揭示不同近似贝叶斯算法间的性能差异。
原文摘要 · Abstract (English)
Variational inference (VI) is widely used for approximate inference in Bayesian machine learning. In addition to this practical success, generalization bounds for variational inference and related algorithms have been developed, mostly through the connection to PAC-Bayes analysis. A second line of work has provided algorithm-specific generalization bounds through stability arguments or using mutual information bounds, and has shown that the bounds are tight in practice, but unfortunately these bounds do not directly apply to approximate Bayesian algorithms. This paper fills this gap by developing algorithm-specific stability based generalization bounds for a class of approximate Bayesian algorithms that includes VI, specifically when using stochastic gradient descent to optimize their objective. As in the non-Bayesian case, the generalization error is bounded by by expected parameter differences on a perturbed dataset. The new approach complements PAC-Bayes analysis and can provide tighter bounds in some cases. An experimental illustration shows that the new approach yields non-vacuous bounds on modern neural network architectures and datasets and that it can shed light on performance differences between variant approximate Bayesian algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。