改进高样本效率强化学习，提升稳定性和训练效果
Reparameterization Proximal Policy Optimization
- 利用时间反向传播统一在线与离线更新机制
- 在多种任务上保持高样本效率并超越现有方法
- 适合需要高效稳定训练的强化学习研究者
通过可微分动态模型，重参数化策略梯度（RPG）实现了高样本效率。然而，当前方法存在两大缺陷：计算成本高的动态雅可比矩阵利用率低，以及固有的训练不稳定性。虽然样本复用可缓解前者问题，但缺乏理论指导，盲目尝试可能加剧后者。为此，本文提出重参数化近端策略优化（RPO）。我们证明，在样本复用下，RPG可通过反向传播时间自然优化类似PPO的代理目标，建立了在线与离线更新的统一框架。为进一步保证稳定性，RPO引入针对RPG设计的截断策略梯度机制，并采用显式的KL散度正则化。实验表明，RPO在多样任务中均保持优异样本效率，性能持续优于或达到当前最优水平。
原文摘要 · Abstract (English)
By leveraging differentiable dynamics, Reparameterization Policy Gradient (RPG) achieves high sample efficiency. However, current approaches are hindered by two critical limitations: the under-utilization of computationally expensive dynamics Jacobians and inherent training instability. While sample reuse offers a remedy for under-utilization, no prior principled framework exists, and naive attempts risk exacerbating instability. To address these challenges, we propose Reparameterization Proximal Policy Optimization (RPO). We first establish that under sample reuse, RPG naturally optimizes a PPO-style surrogate objective via Backpropagation Through Time, providing a unified framework for both on- and off-policy updates. To further ensure stability, RPO integrates a clipped policy gradient mechanism tailored for RPG and employs explicit Kullback-Leibler divergence regularization. Experimental results demonstrate that RPO maintains superior sample efficiency and consistently outperforms or achieves state-of-the-art performance across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。