用演示数据引导探索噪声，提升离线强化学习样本效率
Reinforcement Learning via Implicit Imitation Guidance
- 仅用演示数据生成探索噪声，不强制模仿具体动作
- 在7个连续控制任务中性能提升2-3倍
- 适合缺乏密集奖励信号的离线强化学习场景
我们研究样本高效的强化学习问题，即在缺乏密集奖励信号时，利用先验数据（如示范）进行初始化。传统方法通过模仿学习目标正则化或获取参考策略，但可能损害长期性能，因与奖励最大化不直接对齐。本文提出仅用先验数据通过向策略添加噪声来引导探索，避免显式行为克隆约束。核心思想是：示范数据最有效的用途是识别应探索的动作，而非强制执行特定动作。所提方法在7个模拟连续控制任务中，相比现有离线强化学习方法性能提升2-3倍。
原文摘要 · Abstract (English)
We study the problem of sample efficient reinforcement learning, where prior data such as demonstrations are provided for initialization in lieu of a dense reward signal. A natural approach is to incorporate an imitation learning objective, either as regularization during training or to acquire a reference policy. However, imitation learning objectives can ultimately degrade long-term performance, as it does not directly align with reward maximization. In this work, we propose to use prior data solely for guiding exploration via noise added to the policy, sidestepping the need for explicit behavior cloning constraints. The key insight in our framework, Data-Guided Noise (DGN), is that demonstrations are most useful for identifying which actions should be explored, rather than forcing the policy to take certain actions. Our approach achieves up to 2-3x improvement over prior reinforcement learning from offline data methods across seven simulated continuous control tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。