自适应步长让贝叶斯神经网络采样更准更快
Adaptive Stepsizing for Stochastic Gradient Langevin Dynamics in Bayesian Neural Networks
- 根据梯度大小动态调整步长，高曲率处变小、平坦处变大
- 在二维测试和图像分类任务中，后验采样精度优于传统方法
- 无需额外校正项，适合追求高效准确的贝叶斯学习研究者
贝叶斯神经网络（BNN）需要可扩展的采样算法来近似参数的后验分布。现有随机梯度马尔可夫链蒙特卡洛（SGMCMC）方法对步长选择极为敏感，自适应版本如pSGLD通常需添加代价高昂的发散校正项才能正确采样。本文基于近期提出的`SamAdams'时间步长自适应框架（Leimkuhler, Lohmann, and Whalley 2025），提出SA-SGLD：通过时间重缩放机制，根据监测量（通常是局部梯度范数）调节步长。该方法能在高曲率区域自动缩小步长、平坦区域扩大步长，提升稳定性和混合效率且无偏差。实验表明，相比SGLD，SA-SGLD在高曲率二维模型及使用尖锐先验的图像分类任务中实现更精确的后验采样。
原文摘要 · Abstract (English)
Bayesian neural networks (BNNs) require scalable sampling algorithms to approximate posterior distributions over parameters. Existing stochastic gradient Markov Chain Monte Carlo (SGMCMC) methods are highly sensitive to the choice of stepsize and adaptive variants such as pSGLD typically fail to sample the correct invariant measure without addition of a costly divergence correction term. In this work, we build on the recently proposed `SamAdams' framework for timestep adaptation (Leimkuhler, Lohmann, and Whalley 2025), introducing an adaptive scheme: SA-SGLD, which employs time rescaling to modulate the stepsize according to a monitored quantity (typically the local gradient norm). SA-SGLD can automatically shrink stepsizes in regions of high curvature and expand them in flatter regions, improving both stability and mixing without introducing bias. We show that our method can achieve more accurate posterior sampling than SGLD on high-curvature 2D toy examples and in image classification with BNNs using sharp priors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。