基于沃瑟斯坦几何的新型强化学习算法，无需计算策略对数密度。
Wasserstein Proximal Policy Gradient
- 用最优传输与热扩散交替更新，避免计算策略梯度
- 理论证明全局线性收敛，支持隐式策略直接建模
- 代码简洁，连续控制任务表现优异，适合复杂策略建模
本文从沃瑟斯坦几何视角研究连续动作、熵正则化强化学习的策略梯度方法。基于沃瑟斯坦邻近更新，通过算子分裂框架交替执行最优传输与高斯卷积热步，推导出沃瑟斯坦邻近策略梯度（WPPG）。该方法无需计算策略对数密度及其梯度，可直接应用于以前向映射定义的表达性强的隐式随机策略。理论上，WPPG在精确策略评估和带可控近似误差的演员-评论家实现中均具有全局线性收敛速率。实验表明，WPPG实现简单，在标准连续控制基准上性能具竞争力。
原文摘要 · Abstract (English)
We study policy gradient methods for continuous-action, entropy-regularized reinforcement learning through the lens of Wasserstein geometry. Starting from a Wasserstein proximal update, we derive Wasserstein Proximal Policy Gradient (WPPG) via an operator-splitting scheme that alternates an optimal transport update with a heat step implemented by Gaussian convolution. This formulation avoids evaluating the policy's log density or its gradient, making the method directly applicable to expressive implicit stochastic policies specified as pushforward maps. We establish a global linear convergence rate for WPPG, covering both exact policy evaluation and actor-critic implementations with controlled approximation error. Empirically, WPPG is simple to implement and attains competitive performance on standard continuous-control benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。