解决强化学习中探索效率低的问题，提升稀疏奖励场景下的学习效果。
KEA: Keeping Exploration Alive by Proactively Coordinating Exploration Strategies
- 用双代理切换机制协调随机策略与新颖性探索
- 在深海任务和深蓝控制套件上显著提升学习效率
- 适合需要高效探索的复杂强化学习任务
软演员-评论家（SAC）在连续控制任务中表现优异，但在稀疏奖励环境下因奖励稀少导致探索效率低下。尽管基于新颖性的探索方法能激励智能体前往新状态，但与SAC的随机策略结合时易引发探索低效和重复采样问题。本文提出KEA（保持探索活力），通过一个新颖性增强的SAC代理与标准SAC代理之间的主动切换机制，实现协同优化。该机制在高新颖性区域维持随机性，提升探索效率并减少冗余样本。我们在2D导航任务中分析该问题，并在DeepSea硬探索基准及DeepMind Control Suite的稀疏奖励控制任务上验证。实验表明，相比当前最优的新颖性探索基线，KEA在稀疏奖励设置下显著提升了学习效率与鲁棒性。
原文摘要 · Abstract (English)
Soft Actor-Critic (SAC) has achieved notable success in continuous control tasks but struggles in sparse reward settings, where infrequent rewards make efficient exploration challenging. While novelty-based exploration methods address this issue by encouraging the agent to explore novel states, they are not trivial to apply to SAC. In particular, managing the interaction between novelty-based exploration and SAC's stochastic policy can lead to inefficient exploration and redundant sample collection. In this paper, we propose KEA (Keeping Exploration Alive) which tackles the inefficiencies in balancing exploration strategies when combining SAC with novelty-based exploration. KEA integrates a novelty-augmented SAC with a standard SAC agent, proactively coordinated via a switching mechanism. This coordination allows the agent to maintain stochasticity in high-novelty regions, enhancing exploration efficiency and reducing repeated sample collection. We first analyze this potential issue in a 2D navigation task, and then evaluate KEA on the DeepSea hard-exploration benchmark as well as sparse reward control tasks from the DeepMind Control Suite. Compared to state-of-the-art novelty-based exploration baselines, our experiments show that KEA significantly improves learning efficiency and robustness in sparse reward setups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。