用智能体自身价值估计做奖励设计，加速强化学习训练。
Bootstrapped Reward Shaping
- 用当前状态价值估计作为潜在函数进行奖励塑形
- 在Atari游戏上显著提升训练速度，收敛更快
- 无需人工设计，适合复杂任务的自动奖励优化
在强化学习中，稀疏奖励环境需要大量环境交互才能获得奖励信号。为增加奖励观察频率，提出了基于势函数的奖励塑形（PBRS），可在不改变最优策略的前提下提供更密集的奖励信号。但该方法依赖于任务相关的潜在函数设计，否则会损害训练性能。本文提出一种自举式奖励塑形方法（BSRS），将智能体当前的状态价值函数估计作为PBRS的潜在函数。我们在表格设置下提供了收敛性证明，深入分析了深度强化学习中的训练动态，并在Atari基准测试中验证了该方法能显著提升训练速度。
原文摘要 · Abstract (English)
In reinforcement learning, especially in sparse-reward domains, many environment steps are required to observe reward information. In order to increase the frequency of such observations, "potential-based reward shaping" (PBRS) has been proposed as a method of providing a more dense reward signal while leaving the optimal policy invariant. However, the required "potential function" must be carefully designed with task-dependent knowledge to not deter training performance. In this work, we propose a "bootstrapped" method of reward shaping, termed BSRS, in which the agent's current estimate of the state-value function acts as the potential function for PBRS. We provide convergence proofs for the tabular setting, give insights into training dynamics for deep RL, and show that the proposed method improves training speed in the Atari suite.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。