通过自动分割状态空间，提升强化学习在稀疏奖励环境下的探索效率
Autonomous state-space segmentation for Deep-RL sparse reward scenarios
- 构建双阶段架构：先内在驱动探索，再目标导向学习
- 在不依赖外在奖励的超级马里奥环境中实现成功通关
- 自动生成子目标，避免重复探索已知路径
在开放式自主学习场景中，处理稀疏奖励环境始终是关键挑战。内在动机可有效辅助深度强化学习算法在此类场景中学习。例如，新颖性或好奇心等内在奖励信号通常用于在外部奖励延迟或缺失时增强探索。本文基于先前工作，提出一种两级架构：交替进行内在驱动的探索与自主子目标生成，随后进入稀疏奖励的目标导向策略学习。核心思想是构建多个小型专用网络，分别负责特定子路径，并作为未来探索的起点，无需从头重新探索已知路径。该系统在Gym SuperMarioBros环境中训练并测试，未使用任何额外外在奖励。结果验证了该方法的有效性，凸显了自动分割环境以生成高效通向最终目标路径的重要性。
原文摘要 · Abstract (English)
Dealing with environments with sparse rewards has always been crucial for systems developed to operate in autonomous open-ended learning settings. Intrinsic Motivations could be an effective way to help Deep Reinforcement Learning algorithms learn in such scenarios. In fact, intrinsic reward signals, such as novelty or curiosity, are generally adopted to improve exploration when extrinsic rewards are delayed or absent. Building on previous works, we tackle the problem of learning policies in the presence of sparse rewards by proposing a two-level architecture that alternates an ''intrinsically driven'' phase of exploration and autonomous sub-goal generation, to a phase of sparse reward, goal-directed policy learning. The idea is to build several small networks, each one specialized on a particular sub-path, and use them as starting points for future exploration without the need to further explore from scratch previously learnt paths. Two versions of the system have been trained and tested in the Gym SuperMarioBros environment without considering any additional extrinsic reward. The results show the validity of our approach and the importance of autonomously segment the environment to generate an efficient path towards the final goal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。