为有界支持的q-高斯分布推导新Stein恒等式,降低梯度估计方差。
A Stein Identity for q-Gaussians with Bounded Support
- 基于援引分布简化q-高斯的Bonnet与Price型定理
- 所得梯度估计器方差显著降低,接近高斯情形
- 适用于贝叶斯深度学习与尖锐感知优化场景
Stein恒等式是机器学习中生成模型、随机优化等问题的核心工具,广泛用于高斯分布下期望梯度的计算。然而对非高斯分布的研究较少。本文聚焦有界支持的q-高斯分布,通过扩展Landsman、Vanduffel与Yao(2013)的工作,推导出新的Bonnet型与Price型定理,并利用援引分布简化其形式,从而获得与高斯情形几乎相同结构的梯度估计器,易于实现。实验表明,有界支持分布可有效降低梯度估计方差,对贝叶斯深度学习和尖锐感知最小化具有潜在价值。整体上,本工作显著简化了该类重要非高斯分布上的Stein恒等式应用。
原文摘要 · Abstract (English)
Stein's identity is a fundamental tool in machine learning with applications in generative models, stochastic optimization, and other problems involving gradients of expectations under Gaussian distributions. Less attention has been paid to problems with non-Gaussian expectations. Here, we consider the class of bounded-support $q$-Gaussians and derive a new Stein identity leading to gradient estimators which have nearly identical forms to the Gaussian ones, and which are similarly easy to implement. We do this by extending the previous results of Landsman, Vanduffel, and Yao (2013) to prove new Bonnet- and Price-type theorems for q-Gaussians. We also simplify their forms by using escort distributions. Our experiments show that bounded-support distributions can reduce the variance of gradient estimators, which can potentially be useful for Bayesian deep learning and sharpness-aware minimization. Overall, our work simplifies the application of Stein's identity for an important class of non-Gaussian distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。