用少量成功演示加速稀疏奖励下的强化学习
Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings
- 用离线演示预计算价值估计，作为早期学习目标
- 在基准任务上显著加快收敛，提升样本效率
- 适合演示数据少或质量不高的强化学习场景
稀疏奖励环境中的强化学习因缺乏有效反馈而面临重大挑战。本文提出一种简单但高效的方法：利用少量成功演示初始化智能体的价值函数。通过离线计算演示中的价值估计，并将其作为早期学习的目标，该方法为智能体提供了对潜在有效动作的有用先验。随后，智能体通过标准在线交互进一步优化这些估计。这种从离线到在线的混合范式显著减轻了探索负担，在稀疏奖励设置下大幅提升了样本效率。在基准任务上的实验表明，该方法即使在演示数据极少或质量不佳的情况下，也能加速收敛并优于标准基线。
原文摘要 · Abstract (English)
Reinforcement learning (RL) in sparse-reward environments remains a significant challenge due to the lack of informative feedback. We propose a simple yet effective method that uses a small number of successful demonstrations to initialize the value function of an RL agent. By precomputing value estimates from offline demonstrations and using them as targets for early learning, our approach provides the agent with a useful prior over promising actions. The agent then refines these estimates through standard online interaction. This hybrid offline-to-online paradigm significantly reduces the exploration burden and improves sample efficiency in sparse-reward settings. Experiments on benchmark tasks demonstrate that our method accelerates convergence and outperforms standard baselines, even with minimal or suboptimal demonstration data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。