基于最优传输设计新策略优化算法，兼顾确定性与随机策略优势。
Wasserstein Policy Optimization
- 用最优传输梯度流推导出可闭式更新的策略优化方法
- 在连续控制任务上优于当前主流方法，包括磁约束聚变控制任务
- 无需重参数化技巧，支持任意动作分布的随机策略
我们提出 Wasserstein 策略优化（WPO），一种用于连续动作空间强化学习的演员-评论家算法。WPO 可被看作是在所有策略空间上对 Wasserstein 梯度流进行投影至有限维参数空间（如神经网络权重）后的近似，从而得到一个简单且完全通用的闭式更新规则。该算法结合了确定性策略梯度与经典策略梯度的诸多优点：与确定性策略梯度类似，它利用动作价值函数关于动作的梯度信息；与经典策略梯度类似，它可应用于任意动作分布的随机策略，而无需使用重参数化技巧。我们在 DeepMind Control Suite 和磁约束聚变控制任务上进行了实验，结果表明其性能优于当前最先进的连续控制方法。
原文摘要 · Abstract (English)
We introduce Wasserstein Policy Optimization (WPO), an actor-critic algorithm for reinforcement learning in continuous action spaces. WPO can be derived as an approximation to Wasserstein gradient flow over the space of all policies projected into a finite-dimensional parameter space (e.g., the weights of a neural network), leading to a simple and completely general closed-form update. The resulting algorithm combines many properties of deterministic and classic policy gradient methods. Like deterministic policy gradients, it exploits knowledge of the gradient of the action-value function with respect to the action. Like classic policy gradients, it can be applied to stochastic policies with arbitrary distributions over actions -- without using the reparameterization trick. We show results on the DeepMind Control Suite and a magnetic confinement fusion task which compare favorably with state-of-the-art continuous control methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。