将随机梯度下降与贝叶斯后验校正结合，提升训练速度。
SVRG and Beyond via Posterior Correction
- 用后验校正视角重推SVRG，发现其为高斯后验的特例。
- 基于更灵活分布扩展出新算法，含牛顿型和自适应变体。
- 首次建立SVRG与贝叶斯方法的深层联系，适合优化研究者。
随机方差减少梯度（SVRG）及其变体通过梯度修正加速训练，虽已提出十余年，但从未从贝叶斯基础层面被理解。本文填补这一空白，揭示了SVRG与近期提出的贝叶斯方法‘后验校正’之间的意外关联。核心贡献在于证明SVRG可视为各向同性高斯后验下的后验校正特例。通过采用更灵活的指数族后验，可自然导出新的SVRG扩展。我们由此推导出两种新算法：一种是带新海森修正的类牛顿变体，另一种是可扩展至大规模问题的类Adam变体。本工作首次将SVRG与贝叶斯框架连接，并利用其加速训练。
原文摘要 · Abstract (English)
Stochastic Variance Reduced Gradient (SVRG) and its variants aim to speed-up training by using gradient corrections. Originally proposed over a decade ago, these methods have never been connected to any Bayesian method at a fundamental level. Here, we fill this gap and derive surprising new connections of SVRG to a recently proposed Bayesian method called `posterior correction'. Our main contribution is to show that SVRG can be recovered as a special case of posterior-correction over isotropic-Gaussian posteriors. Novel extensions of SVRG are automatically obtained by using more flexible exponential-family posteriors. We derive two new such extensions by using Gaussian families: a Newton-like variant with novel Hessian corrections, and an Adam-like extension that scales to large problems. Our work is the first to connect SVRG to Bayes and use it to speed-up training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。