用BiGAN检测状态新颖性，提升强化学习探索效率。
Adventurer: Exploration with BiGAN for Deep Reinforcement Learning
- 用双向生成对抗网络估计高维观测下的状态新颖性
- 在Mujoco和Atari任务上达到与顶尖算法相当的性能
- 适合处理复杂视觉输入的强化学习场景
深度强化学习在解决复杂问题方面已取得显著进展，但样本效率和局部最优仍是主要挑战。为此，基于新颖性的探索策略应运而生，展现出良好前景。然而，现有方法在高维复杂观测任务中表现不佳，且无一种算法在所有任务上均胜出。本文提出Adventurer，一种基于双向生成对抗网络（BiGAN）的新颖性驱动探索算法，利用训练好的生成器对已访问状态分布建模，通过重构误差判断新状态：若输入状态为新状态，则其从特定隐变量重构时会产生更大误差。该方法能有效估计复杂观测下的状态新颖性，并可与内在奖励机制结合。实验表明，Adventurer在多种主流基准任务上表现优异，涵盖连续机器人操控（如Mujoco机器人）和高维图像任务（如Atari游戏），结果具有竞争力。
原文摘要 · Abstract (English)
Recent developments in deep reinforcement learning have been very successful in learning complex, previously intractable problems. Sample efficiency and local optimality, however, remain significant challenges. To address these challenges, novelty-driven exploration strategies have emerged and shown promising potential. Unfortunately, no single algorithm outperforms all others in all tasks and most of them struggle with tasks with high-dimensional and complex observations. In this work, we propose Adventurer, a novelty-driven exploration algorithm that is based on Bidirectional Generative Adversarial Networks (BiGAN), where BiGAN is trained to estimate state novelty. Intuitively, a generator that has been trained on the distribution of visited states should only be able to generate a state coming from the distribution of visited states. As a result, novel states using the generator to reconstruct input states from certain latent representations would lead to larger reconstruction errors. We show that BiGAN performs well in estimating state novelty for complex observations. This novelty estimation method can be combined with intrinsic-reward-based exploration. Our empirical results show that Adventurer produces competitive results on a range of popular benchmark tasks, including continuous robotic manipulation tasks (e.g. Mujoco robotics) and high-dimensional image-based tasks (e.g. Atari games).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。