用事后目标重加权加速PPO在稀疏奖励环境中的学习
Hindsight Experience Replay Accelerates Proximal Policy Optimization
- 将经验回放的目标改为轨迹中曾到达的任意状态
- 在捕食者-猎物环境中使PPO训练速度提升数倍
- 首次证明赫尔机制可有效用于PPO这类在线策略算法
事后经验回放（HER）通过在事后将任务目标替换为轨迹中实际达到的状态，加速稀疏奖励环境下的离线策略强化学习。由于这种事后修改违反了在线策略算法的假设，传统上HER不适用于在线策略算法。本文首次表明，在自定义的捕食者-猎物环境中，将HER应用于近端策略优化（PPO）可显著加速其学习过程。
原文摘要 · Abstract (English)
Hindsight experience replay (HER) accelerates off-policy reinforcement learning algorithms for environments that emit sparse rewards by modifying the goal of the episode post-hoc to be some state achieved during the episode. Because post-hoc modification of the observed goal violates the assumptions of on-policy algorithms, HER is not typically applied to on-policy algorithms. Here, we show that HER can dramatically accelerate proximal policy optimization (PPO), an on-policy reinforcement learning algorithm, when tested on a custom predator-prey environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。