用集成误差设计奖励，让智能体首次访问状态时就有探索激励。
Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learning
- 维护多个随机动作值函数，用其预测误差生成探索奖励。
- 在经典环境上优于Bootstrap DQN、RND和ACB，Atari上也表现良好。
- 适合需要深度探索的强化学习任务，尤其新状态首次访问场景。
乐观的价值估计是强化学习中一种定向探索机制。智能体根据价值估计加上价值奖励进行贪心行动。该奖励可通过在奖励奖励上估计价值函数，并传播奖励周围的局部不确定性来学习。然而，此方法仅在已观察到更高奖励奖励后，才对某状态-动作对增加价值奖励，无法激励智能体首次访问该状态-动作。本文提出一种名为基于集成误差的价值奖励（VBE）的探索算法，维护一组随机动作值函数（RQFs），利用这些RQFs的预测误差设计能提供首次访问乐观性和深层探索的价值奖励。关键思想是设计这些RQFs的奖励，使得价值奖励可降至零。实验表明，VBE在多个经典探索测试环境中优于Bootstrap DQN及两种奖励奖励方法（RND和ACB），并在Atari等复杂环境中展现出良好的可扩展性。
原文摘要 · Abstract (English)
Optimistic value estimates provide one mechanism for directed exploration in reinforcement learning (RL). The agent acts greedily with respect to an estimate of the value plus what can be seen as a value bonus. The value bonus can be learned by estimating a value function on reward bonuses, propagating local uncertainties around rewards. However, this approach only increases the value bonus for an action retroactively, after seeing a higher reward bonus from that state and action. Such an approach does not encourage the agent to visit a state and action for the first time. In this work, we introduce an algorithm for exploration called Value Bonuses with Ensemble errors (VBE), that maintains an ensemble of random action-value functions (RQFs). VBE uses the errors in the estimation of these RQFs to design value bonuses that provide first-visit optimism and deep exploration. The key idea is to design the rewards for these RQFs in such a way that the value bonus can decrease to zero. We show that VBE outperforms Bootstrap DQN and two reward bonus approaches (RND and ACB) on several classic environments used to test exploration and provide demonstrative experiments that it can scale easily to more complex environments like Atari.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。