无需奖励信号,通过最大化状态覆盖预训练探索策略。
Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization

- 以状态空间覆盖率为目标,用熵最大化实现无奖赏预训练。
- 在稀疏导航任务中,状态覆盖更均匀,下游任务初始化更强。
- 适合多任务、持续学习等无奖赏场景,可快速适配新任务。
稀疏奖励是强化学习的核心挑战,因智能体直到抵达目标才获得有效信号。内在奖励方法通过优化新颖性、预测误差或技能多样性等非平稳目标来引入监督信号,但通常需能评估外部(稀疏)奖励,无论是在线还是离线重标记存储的转移数据。这一限制在多任务、元学习和持续强化学习中尤为棘手,因智能体与环境交互通常是无奖励的。本文提出一种方法,预训练可迁移的探索策略,使智能体能在下游任务中快速适应稀疏奖励。其目标为最大化状态空间的占用率覆盖,可表述为熵最大化。算法实现ROVER利用最近在强化学习算子形式上的进展,通过学习的求解器世界模型估计占用率,绕过密度与熵估计的常见难题。ROVER进一步引入虚拟“汇点”状态,平衡已知状态覆盖与未探索区域扩展,防止学习过程中的循环膨胀-坍缩行为。在表格和基于像素的稀疏导航任务中,相比标准无奖赏基线,ROVER实现了更均匀的累积覆盖,并为下游任务提供了更强的初始化。
原文摘要 · Abstract (English)
Sparse rewards pose a central challenge in reinforcement learning, since agents receive no informative signal until they reach their goal. Intrinsic-reward methods address this issue by optimizing non-stationary objectives such as novelty, prediction error, or skill diversity, thereby injecting a supervision signal into the problem. While effective, these methods often require that the extrinsic (sparse) reward can be evaluated -- either online or during offline relabeling of the stored transitions. This limitation is particularly vexing for multi-task, meta-, and continual reinforcement learning, where agents' interactions with the environment are usually reward-free. In this work, we present a method to pre-train transferable exploration policies that rapidly adapt to sparse rewards at downstream task time. Our objective maximizes state-space covering for the occupancy measure, and can be framed in terms of entropy maximization. Its algorithmic implementation, ROVER, leverages recent advances on the operatorial formulation of RL to estimate occupancy with a learned resolvent world model, bypassing common hurdles associated with density and entropy estimation. ROVER further introduces a virtual "sink" state for unexplored regions, balancing coverage of known states with expansion into unseen ones and preventing cyclic expansion-collapse behavior during learning. In tabular and pixel-based sparse navigation tasks, ROVER produces more uniform aggregate coverage and stronger initializations for downstream tasks than standard reward-free baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。