改进强化学习经验回放，让智能体在稀疏奖励的复杂迷宫中更高效探索。
Generalized Back-Stepping Experience Replay in Sparse-Reward Environments
- 引入重标注机制与多样化采样策略，扩展原方法至稀疏奖励环境。
- 在多种具有高度对称结构的迷宫中，性能和稳定性显著提升。
- 适合需要复杂探索的稀疏奖励任务，如迷宫导航、机器人路径规划。
回溯式经验回放（BER）是一种可加速可逆环境学习效率的强化学习技术，通过使用收集经验生成的回溯转移与正常前向转移联合训练智能体。然而，原始算法仅适用于无需复杂探索的密集奖励环境，限制了其潜力发挥。本文提出一种增强版BER——广义回溯式经验回放（GBER），将其扩展至稀疏奖励环境，尤其适用于具有复杂结构需探索的任务。GBER通过引入重标注机制并采用多样化采样策略提升性能。我们在基于目标条件深度确定性策略梯度的离线学习算法基础上进行测试，涵盖多种迷宫导航环境。实验结果表明，相较于基线算法，GBER在各类稀疏奖励环境中均显著提升了性能与稳定性，尤其在具有高度结构对称性的环境中表现突出。
原文摘要 · Abstract (English)
Back-stepping experience replay (BER) is a reinforcement learning technique that can accelerate learning efficiency in reversible environments. BER trains an agent with generated back-stepping transitions of collected experiences and normal forward transitions. However, the original algorithm is designed for a dense-reward environment that does not require complex exploration, limiting the BER technique to demonstrate its full potential. Herein, we propose an enhanced version of BER called Generalized BER (GBER), which extends the original algorithm to sparse-reward environments, particularly those with complex structures that require the agent to explore. GBER improves the performance of BER by introducing relabeling mechanism and applying diverse sampling strategies. We evaluate our modified version, which is based on a goal-conditioned deep deterministic policy gradient offline learning algorithm, across various maze navigation environments. The experimental results indicate that the GBER algorithm can significantly boost the performance and stability of the baseline algorithm in various sparse-reward environments, especially those with highly structural symmetricity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。