通过最大化信息增益引导探索，提升强化学习性能。
MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization
- 以信息增益为内在奖励，引导智能体探索更有价值的状态转移。
- 在多臂赌博机中实现次线性遗憾，理论证明探索效率更高。
- 适用于复杂视觉控制任务，显著优于现有方法的探索能力。
强化学习需在利用当前最优策略与探索潜在高回报选项间取得平衡。多数算法采用无向探索（随机动作序列），也可通过内在奖励（如好奇心或模型不确定性）实现有向探索。但有效平衡任务奖励与内在奖励颇具挑战，且常依赖具体任务。本文提出MaxInfoRL框架，通过最大化对底层任务的信息增益来引导探索。结合Boltzmann探索，该方法自然权衡价值函数最大化与状态、奖励、动作熵的最大化。我们在简化版多臂赌博机设定中证明该方法可实现次线性遗憾。随后将此通用框架应用于多种离策略无模型强化学习算法，处理连续状态-动作空间，得到新算法，在困难探索问题及复杂视觉控制任务中均表现出更优性能。
原文摘要 · Abstract (English)
Reinforcement learning (RL) algorithms aim to balance exploiting the current best strategy with exploring new options that could lead to higher rewards. Most common RL algorithms use undirected exploration, i.e., select random sequences of actions. Exploration can also be directed using intrinsic rewards, such as curiosity or model epistemic uncertainty. However, effectively balancing task and intrinsic rewards is challenging and often task-dependent. In this work, we introduce a framework, MaxInfoRL, for balancing intrinsic and extrinsic exploration. MaxInfoRL steers exploration towards informative transitions, by maximizing intrinsic rewards such as the information gain about the underlying task. When combined with Boltzmann exploration, this approach naturally trades off maximization of the value function with that of the entropy over states, rewards, and actions. We show that our approach achieves sublinear regret in the simplified setting of multi-armed bandits. We then apply this general formulation to a variety of off-policy model-free RL methods for continuous state-action spaces, yielding novel algorithms that achieve superior performance across hard exploration problems and complex scenarios such as visual control tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。