arXiv:2502.03988cs.LG2025-02中稿 · CIKM '25

提出更通用的Jensen差距界,助力变分推断精度提升

Tight Bounds for Jensen's Gap with Applications to Variational Inference

  • 构建适用于多种函数与分布的广义Jensen差距上界和下界
  • 在对数函数场景下改进变分推断中的变分差距估计
  • 关联PAC-Bayes框架,为概率模型泛化性能提供新视角

自提出以来,Jensen不等式在数学、统计学和机器学习中发挥基础作用,其概率形式揭示了所谓Jensen差距(即凸函数期望与期望点处函数值之差)的非负性。当函数为对数函数时尤为重要,因这支撑了变分推断的诸多应用,此时也常称作变分差距。近期研究聚焦于估计该差距大小,并在不同函数与分布假设下建立紧致的上下界,以应对图模型(如变分自编码器)中似然函数不可计算的实际挑战。本文提出新的、通用的Jensen差距界,适用于广泛的函数与随机变量假设,特别关注指数与对数情形。我们提供了分析与实验双重证据验证方法性能。此外,将我们的界与PAC-Bayes框架关联,为概率模型的泛化性能带来新见解。

原文摘要 · Abstract (English)

Since its original formulation, Jensen's inequality has played a fundamental role across mathematics, statistics, and machine learning, with its probabilistic version highlighting the nonnegativity of the so-called Jensen's gap, i.e., the difference between the expectation of a convex function and the function at the expectation. Of particular importance is the case when the function is logarithmic, as this setting underpins many applications in variational inference, where the term variational gap is often used interchangeably. Recent research has focused on estimating the size of Jensen's gap and establishing tight lower and upper bounds under various assumptions on the underlying function and distribution, driven by practical challenges such as the intractability of log-likelihood in graphical models like variational autoencoders (VAEs). In this paper, we propose new, general bounds for Jensen's gap that accommodate a broad range of assumptions on both the function and the random variable, with special attention to exponential and logarithmic cases. We provide both analytical and empirical evidence for the performance of our method. Furthermore, we relate our bounds to the PAC-Bayes framework, providing new insights into generalization performance in probabilistic models.

变分推断概率建模不等式边界泛化性能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。