通过不确定性驱动的批评者学习，提升连续控制中的探索效率。
Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
- 用分布式Langevin蒙特卡洛实现近似Thompson采样，优化批评者。
- 在多个后验模式间并行探索，提升策略多样性。
- 适合追求高效探索的连续控制研究者使用。
现有无模型强化学习算法在连续控制任务中因缺乏系统性探索机制而样本效率低下。受Thompson采样成功启发，本文提出一种新算法Langevin Soft Actor-Critic（LSAC），将提升批评者学习的不确定性估计置于政策优化之上。LSAC引入三项创新：基于分布式Langevin蒙特卡洛（LMC）的$Q$函数更新以实现近似Thompson采样；通过平行退火技术探索$Q$函数后验分布的多个模态；利用$Q$动作梯度正则化生成的扩散状态-动作样本。大量实验表明，LSAC在连续控制任务中性能优于或匹配主流无模型算法。值得注意的是，它是首个在具有连续动作空间的连续控制任务中成功应用基于LMC的Thompson采样的方法。
原文摘要 · Abstract (English)
Existing actor-critic algorithms, which are popular for continuous control reinforcement learning (RL) tasks, suffer from poor sample efficiency due to lack of principled exploration mechanism within them. Motivated by the success of Thompson sampling for efficient exploration in RL, we propose a novel model-free RL algorithm, Langevin Soft Actor Critic (LSAC), which prioritizes enhancing critic learning through uncertainty estimation over policy optimization. LSAC employs three key innovations: approximate Thompson sampling through distributional Langevin Monte Carlo (LMC) based $Q$ updates, parallel tempering for exploring multiple modes of the posterior of the $Q$ function, and diffusion synthesized state-action samples regularized with $Q$ action gradients. Our extensive experiments demonstrate that LSAC outperforms or matches the performance of mainstream model-free RL algorithms for continuous control tasks. Notably, LSAC marks the first successful application of an LMC based Thompson sampling in continuous control tasks with continuous action spaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。