用非精选数据提升强化学习采样效率，解决数据分布偏移问题。
Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
- 通过经验回放和执行引导缓解数据分布偏移
- 在72个视觉运动任务中样本效率接近翻倍
- 适合资源有限的机器人控制场景
利用离线数据是提升在线强化学习（RL）样本效率的有前景方法。本文扩展了可用于离线到在线RL的数据池，引入大量无标注、无奖励、质量混杂且跨多种机器人形态收集的非精选数据。尽管学习世界模型看似适用于此类数据，但我们发现直接微调无法在多数任务上加速RL训练。经深入分析，归因于微调过程中离线与在线数据之间的分布偏移。为此，我们提出两种技术：经验回放与执行引导。结合这两项改进后，非精选离线数据显著提升了RL的样本效率。在有限样本预算下，本方法在覆盖6种机器人形态的72个视觉运动任务中，平均得分接近从零开始基线的两倍；在行走和机器人操作等挑战性任务上，也明显优于以往使用离线数据的方法。
原文摘要 · Abstract (English)
Leveraging offline data is a promising way to improve the sample efficiency of online reinforcement learning (RL). This paper expands the pool of usable data for offline-to-online RL by leveraging abundant non-curated data that is reward-free, of mixed quality, and collected across multiple embodiments. Although learning a world model appears promising for utilizing such data, we find that naive fine-tuning fails to accelerate RL training on many tasks. Through careful investigation, we attribute this failure to the distributional shift between offline and online data during fine-tuning. To address this issue and effectively use the offline data, we propose two techniques: \emph{i)} experience rehearsal and \emph{ii)} execution guidance. With these modifications, the non-curated offline data substantially improves RL's sample efficiency. Under limited sample budgets, our method achieves nearly twice the aggregate score of learning-from-scratch baselines across 72 visuomotor tasks spanning 6 embodiments. On challenging tasks such as locomotion and robotic manipulation, it outperforms prior methods that utilize offline data by a decent margin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。