用扩散模型从能量策略中采样，提升连续控制的样本效率和多模态建模能力。
Sampling from Energy-based Policies using Diffusion
- 以负Q值为能量函数，构建扩散过程实现能量策略采样。
- 在连续控制任务中提升样本效率，有效捕捉多峰动作分布。
- 适合需要复杂策略建模的强化学习场景，如机器人控制。
能量基策略为强化学习中的复杂多模态行为建模提供了灵活框架。在最大熵强化学习中，最优策略是基于软Q函数的玻尔兹曼分布，但在连续动作空间中直接采样该分布计算上不可行。因此，现有方法通常采用高斯等简单参数化分布表示策略，限制了对多模态动作分布的建模能力。本文提出一种基于扩散的方法，以负Q函数作为能量函数，实现能量基策略的采样。基于此，我们提出了扩散Q采样（Diffusion Q-Sampling, DQS）的演员-评论家算法，支持更丰富的策略表达，可在多种环境中实现稳定学习。实验表明，该方法提升了连续控制任务中的样本效率，并能有效捕捉多模态行为，解决了现有方法的关键局限。代码已开源：https://github.com/vineetjain96/Diffusion_Q_Sampling.git。
原文摘要 · Abstract (English)
Energy-based policies offer a flexible framework for modeling complex, multimodal behaviors in reinforcement learning (RL). In maximum entropy RL, the optimal policy is a Boltzmann distribution derived from the soft Q-function, but direct sampling from this distribution in continuous action spaces is computationally intractable. As a result, existing methods typically use simpler parametric distributions, like Gaussians, for policy representation -- limiting their ability to capture the full complexity of multimodal action distributions. In this paper, we introduce a diffusion-based approach for sampling from energy-based policies, where the negative Q-function defines the energy function. Based on this approach, we propose an actor-critic method called Diffusion Q-Sampling (DQS) that enables more expressive policy representations, allowing stable learning in diverse environments. We show that our approach enhances sample efficiency in continuous control tasks and captures multimodal behaviors, addressing key limitations of existing methods. Code is available at https://github.com/vineetjain96/Diffusion_Q_Sampling.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。