用过往成功经验指导探索，提升强化学习样本效率。
Match or Replay: Self Imitating Proximal Policy Optimization
- 通过自模仿机制复用高奖励状态动作对，优化策略更新
- 在密集奖励环境和稀疏奖励环境中均实现更快收敛
- 适合需要高效探索的复杂强化学习任务
强化学习代理在稀疏奖励环境中常因探索效率低而表现不佳。传统探索策略难以系统性利用已有成功经验，导致样本效率低下。为此，我们提出一种自模仿的在线策略算法，通过复用过去高奖励的状态-动作对来引导策略更新。在密集奖励环境中，利用最优传输距离优先选择与最优轨迹匹配的状态访问分布；在稀疏奖励环境中，均匀重放成功自经历的轨迹以促进结构化探索。在多种环境中的实验表明，该方法显著提升了学习效率：在MuJoCo中表现优异，在部分可观测3D Animal-AI Olympics和多目标PointMaze中也实现了更快收敛和更高成功率，优于当前最先进的自模仿强化学习基线。结果证明自模仿是提升强化学习探索能力的有效策略，适用于更复杂任务。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) agents often struggle with inefficient exploration, particularly in environments with sparse rewards. Traditional exploration strategies can lead to slow learning and suboptimal performance because agents fail to systematically build on previously successful experiences, thereby reducing sample efficiency. To tackle this issue, we propose a self-imitating on-policy algorithm that enhances exploration and sample efficiency by leveraging past high-reward state-action pairs to guide policy updates. Our method incorporates self-imitation by using optimal transport distance in dense reward environments to prioritize state visitation distributions that match the most rewarding trajectory. In sparse-reward environments, we uniformly replay successful self-encountered trajectories to facilitate structured exploration. Experimental results across diverse environments demonstrate substantial improvements in learning efficiency, including MuJoCo for dense rewards and the partially observable 3D Animal-AI Olympics and multi-goal PointMaze for sparse rewards. Our approach achieves faster convergence and significantly higher success rates compared to state-of-the-art self-imitating RL baselines. These findings underscore the potential of self-imitation as a robust strategy for enhancing exploration in RL, with applicability to more complex tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。