通过高置信度回报评估，稳定策略梯度方法的训练过程。
Stabilizing Policy Gradient Methods via Reward Profiling
- 基于高置信度回报选择性更新策略，降低梯度方差。
- 在8个连续控制任务中，收敛速度提升1.5倍，回报方差减少1.75倍。
- 适用于各类策略梯度算法，适合追求稳定高效的RL研究者。
策略梯度方法在近十年来被广泛研究，为强化学习问题提供了高效框架。然而，其性能常不理想，受梯度估计高方差影响，导致奖励改进不可靠且收敛缓慢。本文提出一种通用的回报轮廓化框架，可无缝集成于任意策略梯度算法中，仅基于高置信度性能估计更新策略。理论上证明该方法不会减慢基线算法的收敛速度，但以高概率实现性能的稳定与单调提升。实验上，在八个连续控制基准任务（Box2D和MuJoCo/PyBullet）中，该方法使收敛速度最快提升1.5倍，部分设置下回报方差减少1.75倍。该框架为复杂环境中的可靠高效策略学习提供了一条通用且理论坚实的路径。
原文摘要 · Abstract (English)
Policy gradient methods, which have been extensively studied in the last decade, offer an effective and efficient framework for reinforcement learning problems. However, their performances can often be unsatisfactory, suffering from unreliable reward improvements and slow convergence, due to high variance in gradient estimations. In this paper, we propose a universal reward profiling framework that can be seamlessly integrated with any policy gradient algorithm, where we selectively update the policy based on high-confidence performance estimations. We theoretically justify that our technique will not slow down the convergence of the baseline policy gradient methods, but with high probability, will result in stable and monotonic improvements of their performance. Empirically, on eight continuous-control benchmarks (Box2D and MuJoCo/PyBullet), our profiling yields up to 1.5x faster convergence to near-optimal returns, up to 1.75x reduction in return variance on some setups. Our profiling approach offers a general, theoretically grounded path to more reliable and efficient policy learning in complex environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。