用扩散模型做策略，让强化学习探索更高效。
Maximum Entropy Reinforcement Learning with Diffusion Policy
- 用扩散模型替代高斯策略，捕捉复杂多模态行为分布。
- 在Mujoco上性能优于高斯策略和其他生成模型。
- 适合需要强探索能力的复杂多目标强化学习任务。
软演员-评论家（SAC）算法结合高斯策略已成为实现最大熵强化学习（MaxEnt RL）主流方法，通过最大化熵来促进探索并提升策略鲁棒性。然而，高斯策略在简单任务中表现良好，其固有的单峰特性限制了其在复杂多目标强化学习环境中的探索能力和潜在性能。本文提出一种基于扩散模型的策略表示方法，以实现最大熵强化学习目标，称为最大熵强化学习与扩散策略（MaxEntDP）。该方法能有效进行探索，使策略更接近最优的最大熵策略。在Mujoco基准测试上的实验结果表明，MaxEntDP在性能上超越了高斯策略及其他生成模型，并与当前最先进的基于扩散的在线强化学习算法相当。代码已开源：https://github.com/diffusionyes/MaxEntDP。
原文摘要 · Abstract (English)
The Soft Actor-Critic (SAC) algorithm with a Gaussian policy has become a mainstream implementation for realizing the Maximum Entropy Reinforcement Learning (MaxEnt RL) objective, which incorporates entropy maximization to encourage exploration and enhance policy robustness. While the Gaussian policy performs well on simpler tasks, its exploration capacity and potential performance in complex multi-goal RL environments are limited by its inherent unimodality. In this paper, we employ the diffusion model, a powerful generative model capable of capturing complex multimodal distributions, as the policy representation to fulfill the MaxEnt RL objective, developing a method named MaxEnt RL with Diffusion Policy (MaxEntDP). Our method enables efficient exploration and brings the policy closer to the optimal MaxEnt policy. Experimental results on Mujoco benchmarks show that MaxEntDP outperforms the Gaussian policy and other generative models within the MaxEnt RL framework, and performs comparably to other state-of-the-art diffusion-based online RL algorithms. Our code is available at https://github.com/diffusionyes/MaxEntDP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。