用离线数据自动学出最优目标，让在线强化学习更快更准。
Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
- 通过元学习从离线数据中提炼奖励目标，激发内在探索动机。
- 在四个稀疏奖励的MuJoCo任务中，提升三款算法性能，超越当前最佳方法。
- 适配多种主流强化学习算法，无需调参且计算开销极小。
强化学习在稀疏奖励场景下难以学习最优策略。以往方法通过人工设计的辅助目标结合模仿学习增强在线学习，但当离线数据由非专家策略生成时，会限制策略性能。为此，本文提出广义示范模仿学习（GILD），通过元学习从离线数据中提取知识，生成能引导探索的内在奖励目标。GILD不依赖特定强化学习算法，可灵活适配各类基础离线策略算法；无需领域特定超参数，仅带来极小计算成本增加。在四个具有挑战性的稀疏奖励MuJoCo任务中，三种算法结合GILD后均显著优于现有最先进方法。
原文摘要 · Abstract (English)
A major challenge in Reinforcement Learning (RL) is the difficulty of learning an optimal policy from sparse rewards. Prior works enhance online RL with conventional Imitation Learning (IL) via a handcrafted auxiliary objective, at the cost of restricting the RL policy to be sub-optimal when the offline data is generated by a non-expert policy. Instead, to better leverage valuable information in offline data, we develop Generalized Imitation Learning from Demonstration (GILD), which meta-learns an objective that distills knowledge from offline data and instills intrinsic motivation towards the optimal policy. Distinct from prior works that are exclusive to a specific RL algorithm, GILD is a flexible module intended for diverse vanilla off-policy RL algorithms. In addition, GILD introduces no domain-specific hyperparameter and minimal increase in computational cost. In four challenging MuJoCo tasks with sparse rewards, we show that three RL algorithms enhanced with GILD significantly outperform state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。