提出可全局收敛的变分推断方法,突破传统局部最优瓶颈。
Globally Convergent Variational Inference
- 用前向KL散度+神经网络参数化,实现全局最优解
- 在无限宽网络下证明唯一解存在且梯度下降可收敛
- 实测优于传统ELBO方法,适合复杂后验建模
在变分推断中,通常通过数值优化从分布族中选择后验近似,但以证据下界(ELBO)为目标函数时,仅能保证收敛至局部最优。本文研究一种特定变分推断方法,其本质属于神经后验估计(NPE),通过最小化包含性(前向)KL散度来拟合由神经网络参数化的变分分布。基于神经正切核(NTK)刻画函数空间中的梯度动态,在固定正定NTK的渐近情形下,我们建立条件,使变分目标在再生核希尔伯特空间(RKHS)中存在唯一解,并证明函数空间中的梯度下降可收敛至该唯一函数。消融实验与实际问题表明,该理论可解释有限神经元设置下NPE的行为,且其性能优于常陷入浅层局部最优的ELBO方法。
原文摘要 · Abstract (English)
In variational inference (VI), an approximation of the posterior distribution is selected from a family of distributions through numerical optimization. With the most common variational objective function, known as the evidence lower bound (ELBO), only convergence to a local optimum can be guaranteed. In this work, we instead establish the global convergence of a particular VI method. This VI method, which may be considered an instance of neural posterior estimation (NPE), minimizes an expectation of the inclusive (forward) KL divergence to fit a variational distribution that is parameterized by a neural network. Our convergence result relies on the neural tangent kernel (NTK) to characterize the gradient dynamics that arise from considering the variational objective in function space. In the asymptotic regime of a fixed, positive-definite neural tangent kernel, we establish conditions under which the variational objective admits a unique solution in a reproducing kernel Hilbert space (RKHS). Then, we show that the gradient descent dynamics in function space converge to this unique function. In ablation studies and practical problems, we demonstrate that our results explain the behavior of NPE in non-asymptotic finite-neuron settings, and show that NPE outperforms ELBO-based optimization, which often converges to shallow local optima.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。