解决变分推断中梯度方差无界问题，提升优化稳定性。
SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching
- 采用动态批量与预处理技术改进随机梯度
- 在BG条件下实现收敛性保证，适用于复杂分布族
- 适合需要稳定推断的现代概率建模任务
黑箱变分推断（BBVI）通常依赖随机梯度下降（SGD）优化证据下界（ELBO）。然而，BBVI中的随机梯度天然具有无界方差，违反标准假设，仅满足较弱的Blum-Gladyshev（BG）条件——方差随距离最优解的平方增长。本文弥合了随机优化理论与BBVI实际应用之间的差距。针对参数化分布的广义椭球位置-尺度族，提出两项主要贡献：首先，证明了ELBO解的存在性，这一基础性质在文献中通常被假设；其次，在BG条件下，为带动态批量与预处理的最小批量投影SGD（PSGD）建立了涵盖有限时间与渐近行为的完整收敛性保证。理论框架表明，动态批量结合预处理能系统性地实现严格保证，即使在复杂设置下亦成立。数值实验验证了该方法在现代推断任务中的有效性。
原文摘要 · Abstract (English)
Black-Box Variational Inference (BBVI) typically relies on Stochastic Gradient Descent (SGD) to optimize the Evidence Lower Bound (ELBO). However, the stochastic gradients in BBVI inherently exhibit unbounded variance, violating standard assumptions and instead satisfying the weaker Blum-Gladyshev (BG) condition, where variance grows quadratically with distance from the optimum. In this paper, we bridge the gap between stochastic optimization theory and the practical instances of BBVI. Focusing on the broad elliptic location-scale family of parameterized distributions, we offer two main contributions. First, we prove the existence of an ELBO solution, a foundational property usually assumed a priori in the literature. Second, we establish comprehensive convergence guarantees spanning finite-time and asymptotic regimes for Minibatch Projected SGD (PSGD) equipped with dynamic batching and preconditioning under the BG condition. Our theoretical framework demonstrates that dynamic batching combined with preconditioning systematically enables rigorous guarantees even in complex settings. We illustrate our theoretical findings with numerical results, highlighting the efficacy of our approach for modern inference tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。