用自然梯度优化生成模型，提升训练效率与稳定性
Guiding Time-Varying Generative Models with Natural Gradients on Exponential Family Manifold
- 将生成模型演化投影到指数族流形，建立参数关联
- 无需MCMC即可高效近似KL散度的自然梯度
- 适用于任意指数族参数模型，支持粒子化实现
概率模型的优化在统计学中已有深入研究，但其与生成模型训练的联系仍不充分。本文表明，时变生成模型的演化可投影至指数族流形，从而自然建立生成模型参数与概率模型参数之间的关联。通过沿该流形进行自然梯度下降，实现生成模型的训练。该方法无需依赖蒙特卡洛马尔可夫链（MCMC）即可高效近似不可计算模型的KL散度自然梯度。此外，我们提出了算法的粒子版本，对任意指数族中的参数化模型均具有闭式更新规则。通过简单及真实世界实验验证了所提算法的有效性。代码可在 https://github.com/anewgithubname/iNGD 获取。
原文摘要 · Abstract (English)
Optimising probabilistic models is a well-studied field in statistics. However, its connection with the training of generative models remains largely under-explored. In this paper, we show that the evolution of time-varying generative models can be projected onto an exponential family manifold, naturally creating a link between the parameters of a generative model and those of a probabilistic model. We then train the generative model by moving its projection on the manifold according to the natural gradient descent scheme. This approach also allows us to efficiently approximate the natural gradient of the KL divergence without relying on MCMC for intractable models. Furthermore, we propose particle versions of the algorithm, which feature closed-form update rules for any parametric model within the exponential family. Through toy and real-world experiments, we validate the effectiveness of the proposed algorithms. The code of the proposed algorithms can be found at https://github.com/anewgithubname/iNGD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。