通过保留成功经验提升强化学习采样效率
Optimistic Policy Regularization
- 用动态缓冲区保存高绩效轨迹，引导策略优化方向
- 在49个Atari游戏上，1000万步超越基准方法5000万步表现
- 适用于复杂环境，可提升策略最终性能与训练效率
深度强化学习代理常因过早熵坍缩而放弃探索，未能发现全局最优策略。本文提出乐观策略正则化（OPR），一种轻量级机制，通过维护高绩效轨迹的动态缓冲区，并利用方向性对数比率奖励塑造和辅助行为克隆目标，增强历史成功行为的学习偏好。在近端策略优化（PPO）框架下，OPR显著提升雅达利学习环境（Arcade Learning Environment）的样本效率。在1000万步基准下评估49个Atari游戏，OPR在22个环境中取得最高分，优于标准5000万步基准下的基线方法。此外，OPR在CAGE Challenge 2网络防御挑战中也超越夺冠的Cardiff代理，仅使用相同PPO架构。结果表明，将策略更新锚定于实证成功的轨迹，可同时提升样本效率与最终性能。
原文摘要 · Abstract (English)
Deep reinforcement learning agents frequently suffer from premature convergence, where early entropy collapse causes the policy to discard exploratory behaviors before discovering globally optimal strategies. We introduce Optimistic Policy Regularization (OPR), a lightweight mechanism designed to preserve and reinforce historically successful trajectories during policy optimization. OPR maintains a dynamic buffer of high-performing episodes and biases learning toward these behaviors through directional log-ratio reward shaping and an auxiliary behavioral cloning objective. When instantiated on Proximal Policy Optimization (PPO), OPR substantially improves sample efficiency on the Arcade Learning Environment. Across 49 Atari games evaluated at the 10-million step benchmark, OPR achieves the highest score in 22 environments despite baseline methods being reported at the standard 50-million step horizon. Beyond arcade benchmarks, OPR also generalizes to the CAGE Challenge 2 cyber-defense environment, surpassing the competition-winning Cardiff agent while using the same PPO architecture. These results demonstrate that anchoring policy updates to empirically successful trajectories can improve both sample efficiency and final performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。