改进拉格朗日热浴法稳定性,提升大规模贝叶斯采样精度与效率。
Improving the stability of the covariance-controlled adaptive Langevin thermostat for large-scale Bayesian sampling
- 用截断泰勒展开和缩放平方法数值求解新增项,提升算法稳定性。
- 新方法在大模型上显著降低误差,步长可增大三倍以上。
- 适合需要高精度采样的大规模机器学习场景,如深度神经网络训练。
随机梯度拉格朗日动力学及其变体通过随机子集近似整个数据集的似然,在贝叶斯采样中广泛应用。由于计算效率显著提升,已在大规模机器学习中普及。研究表明,引入噪声力协方差矩阵的协方差控制自适应拉格朗日(CCAdL)热浴法优于主流方法。但其使用移动平均估计协方差矩阵,导致长期趋于常数矩阵,并在数值实验中降低积分器稳定性,限制最大步长。本文提出改进的mCCAdL热浴法:采用缩放平方法的缩放部分结合截断泰勒展开逼近精确解;并设计对称分裂格式替代原方法的欧拉型离散。数值实验表明,新方法在稳定性上大幅优于原CCAdL,且在大规模机器学习任务中显著提升数值精度。
原文摘要 · Abstract (English)
Stochastic gradient Langevin dynamics and its variants approximate the likelihood of an entire dataset, via random (and typically much smaller) subsets, in the setting of Bayesian sampling. Due to the (often substantial) improvement of the computational efficiency, they have been widely used in large-scale machine learning applications. It has been demonstrated that the so-called covariance-controlled adaptive Langevin (CCAdL) thermostat, which incorporates an additional term involving the covariance matrix of the noisy force, outperforms popular alternative methods. A moving average is used in CCAdL to estimate the covariance matrix of the noisy force, in which case the covariance matrix will converge to a constant matrix in long-time limit. Moreover, it appears in our numerical experiments that the use of a moving average could reduce the stability of the numerical integrators, thereby limiting the largest usable stepsize. In this article, we propose a modified CCAdL (i.e., mCCAdL) thermostat that uses the scaling part of the scaling and squaring method together with a truncated Taylor series approximation to the exponential to numerically approximate the exact solution to the subsystem involving the additional term proposed in CCAdL. We also propose a symmetric splitting method for mCCAdL, instead of an Euler-type discretisation used in the original CCAdL thermostat. We demonstrate in our numerical experiments that the newly proposed mCCAdL thermostat achieves a substantial improvement in the numerical stability over the original CCAdL thermostat, while significantly outperforming popular alternative stochastic gradient methods in terms of the numerical accuracy for large-scale machine learning applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。