用强化学习自适应优化马尔可夫链蒙特卡洛采样，提升采样效率。
Harnessing the Power of Reinforcement Learning for Adaptive MCMC
- 将采样器调参转化为强化学习问题，设计基于对比散度的新奖励函数。
- 在posteriodb基准上验证，新方法采样效率显著优于传统策略。
- 适合需要自动调参的复杂概率模型，尤其适合高维数据场景。
采样算法是概率机器学习的核心，近年来工具日益多样，但复杂度上升也带来了更大的调参负担。本文提出将采样器调参视为独立的学习任务。受Wang等（2025）启发，将Metropolis-Hastings算法建模为马尔可夫决策过程，开启利用强化学习（RL）实现自适应调参的可能性。然而，自然奖励如接受率或期望平方跳跃距离对训练信号不足。为此，本文提出一种基于对比散度的新奖励函数，在强化学习马尔可夫链蒙特卡洛（RLMH）框架下表现更优。同时，探索了基于梯度的自适应采样器，平衡转移核灵活性与强化学习可学习性。通过posteriodb基准的综合模拟实验，验证了RLMH的实际有效性。
原文摘要 · Abstract (English)
Sampling algorithms drive probabilistic machine learning, and recent years have seen an explosion in the diversity of tools for this task. However, the increasing sophistication of sampling algorithms is correlated with an increase in the tuning burden. There is now a greater need than ever to treat the tuning of samplers as a learning task in its own right. In a conceptual breakthrough, Wang et al (2025) formulated Metropolis-Hastings as a Markov decision process, opening up the possibility for adaptive tuning using Reinforcement Learning (RL). Their emphasis was on theoretical foundations; realising the practical benefit of Reinforcement Learning Metropolis-Hastings (RLMH) was left for subsequent work. The purpose of this paper is twofold: First, we observe the surprising result that natural choices of reward, such as the acceptance rate, or the expected squared jump distance, provide insufficient signal for training RLMH. Instead, we propose a novel reward based on the contrastive divergence, whose superior performance in the context of RLMH is demonstrated. Second, we explore the potential of RLMH and present adaptive gradient-based samplers that balance flexibility of the Markov transition kernel with learnability of the associated RL task. A comprehensive simulation study using the posteriordb benchmark supports the practical effectiveness of RLMH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。