通过重试机制让策略梯度强化学习自动产生探索行为。
Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
- 用多次采样最大回报期望作为目标，诱导探索自发出现。
- 在MinAtar和Craftax上无需额外奖励即可有效探索。
- 将重试次数连续化，实现对探索强度的精细控制。
在强化学习中，探索带来的收益依赖于相似状态的重复出现：尝试不同动作可提升性能或减少不确定性；若无重复，贪婪策略即为最优。本文提出ReMax目标，通过评估$M$次采样下的期望最大回报（$M$为正整数）并考虑回报不确定性，来衡量策略性能。优化该目标可使随机探索成为涌现属性，无需显式奖励项。为高效优化，我们推导出ReMax的新策略梯度形式，并提出ReMax PPO（RePPO），其将离散重试次数$M$推广至连续参数$m > 0$，实现探索强度的精细调节。实验表明，RePPO在MinAtar和Craftax基准上无需任何显式探索奖励即可有效促进探索。
原文摘要 · Abstract (English)
In reinforcement learning (RL), agents benefit from exploration only because they repeatedly encounter similar states: trying different actions can improve performance or reduce uncertainty; without such retries, a greedy policy is optimal. We formalize this intuition with ReMax, an objective that evaluates a policy by the expected maximum return over $M$ samples, where $M$ is a positive integer, while accounting for return uncertainty. Optimizing this objective induces stochastic exploration as an emergent property, without explicit bonus terms. For efficient policy optimization, we derive a new policy-gradient formulation for ReMax and introduce ReMax PPO (RePPO), a PPO variant that optimizes ReMax while generalizing the discrete retry count $M$ to a continuous parameter $m > 0$, enabling fine-grained control of exploration. Empirically, RePPO promotes exploration, without any explicit exploration bonuses, on the MinAtar and Craftax benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。