用扩散模型生成最优策略轨迹，提升强化学习的多样性与性能。
Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning
- 将扩散模型融入最大熵强化学习，构建可采样的最优策略分布。
- 在连续控制和多模态任务上表现优于或持平基线方法。
- 适配多种强化学习算法，支持灵活扩展与实际应用。
扩散模型擅长从复杂且未归一化的分布中采样。本文将最大熵强化学习（ME-RL）拓展至扩散过程，实现对最优策略轨迹分布的采样。通过最小化扩散策略与最优策略轨迹分布之间反向KL散度的可处理上界，推导出修正的代理目标，并引入扩散增强马尔可夫决策过程（DA-MDP）。DA-MDP 可在不改变原框架前提下，以极小改动集成扩散策略。我们将其应用于近端策略优化（PPO）、Wasserstein策略优化（WPO）和相对熵路径策略优化（REPPO），分别得到 DA-MDP: PPO、DA-MDP: WPO 与 DA-MDP: REPPO。在标准连续控制基准测试中，本方法性能匹配或超越基线；在多模态基准测试中，验证了其建模多模态动作分布的能力。
原文摘要 · Abstract (English)
Diffusion models excel at sampling from complex, unnormalized distributions. In this work, we extend Maximum Entropy Reinforcement Learning (ME-RL) to diffusion processes, enabling sampling from the optimal policy trajectory distribution. By minimizing a tractable upper bound on the reverse KL divergence between the diffusion policy and the optimal policy trajectory distributions, we derive a modified surrogate objective and introduce Diffusion-Augmented Markov Decision Processes (DA-MDPs). DA-MDPs allow for seamless integration of diffusion policies into any ME-RL method with minimal modifications. We demonstrate its effectiveness by adapting Proximal Policy Optimization (PPO), Wasserstein Policy Optimization (WPO), and Relative Entropy Pathwise Policy Optimization (REPPO) into their diffusion-based variants: DA-MDP: PPO, DA-MDP: WPO, and DA-MDP: REPPO. Empirical results on standard continuous-control benchmarks show that our approach matches or outperforms baseline methods, while experiments on multimodal benchmarks confirm its ability to model multimodal action distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。