提出可扩展的随机微正则朗之万采样器,解决大规模贝叶斯推断难题。
Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?
- 设计基于能量方差自适应的梯度噪声预处理机制
- 在高维后验中实现接近最优的采样性能
- 适合大规模贝叶斯神经网络等复杂模型推断
将马尔可夫链蒙特卡洛方法扩展至高维模型仍是贝叶斯深度学习的核心挑战。近期提出的微正则朗之万蒙特卡洛在多种问题上表现优异,但依赖全数据集梯度,难以用于大规模任务。本文首次系统研究其能否有效利用小批量梯度噪声。我们建立了新的连续时间理论分析框架,揭示了两类关键失败模式:由各向异性梯度噪声导致的理论偏差,以及在复杂高维后验中的数值不稳定性。为此,我们提出一种有理论依据的梯度噪声预处理方案,显著降低偏差;并设计了一种基于能量方差的自适应调参器,自动选择步长并动态设定数值保护边界。所提算法在贝叶斯神经网络等高维推断任务中达到当前最优性能。结合近期集成技术,本工作开启了大规模贝叶斯推断的新范式——随机微正则朗之万集成(SMILE)采样器。
原文摘要 · Abstract (English)
Scaling inference methods such as Markov chain Monte Carlo to high-dimensional models remains a central challenge in Bayesian deep learning. A promising recent proposal, microcanonical Langevin Monte Carlo, has shown state-of-the-art performance across a wide range of problems. However, its reliance on full-dataset gradients makes it prohibitively expensive for large-scale problems. This paper addresses a fundamental question: Can microcanonical dynamics effectively leverage mini-batch gradient noise? We provide the first systematic study of this problem, establishing a novel continuous-time theoretical analysis of stochastic-gradient microcanonical dynamics. We reveal two critical failure modes: a theoretically derived bias due to anisotropic gradient noise and numerical instabilities in complex high-dimensional posteriors. To tackle these issues, we propose a principled gradient noise preconditioning scheme shown to significantly reduce this bias and develop a novel, energy-variance-based adaptive tuner that automates step size selection and dynamically informs numerical guardrails. The resulting algorithm is a robust and scalable microcanonical Monte Carlo sampler that achieves state-of-the-art performance on challenging high-dimensional inference tasks like Bayesian neural networks. Combined with recent ensemble techniques, our work unlocks a new class of stochastic microcanonical Langevin ensemble (SMILE) samplers for large-scale Bayesian inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。