对比多种演示增强强化学习方法，发现直接复用数据更有效。
Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
- 将离线演示数据直接用于在线训练，避免复杂预训练
- 行为克隆初始化比复杂离线预训练提升样本效率
- 适合追求高效在线学习的研究者和应用开发者
为提升在线强化学习的样本效率，已有方法利用离线收集的演示数据。这些方法或直接将离线数据作为训练样本，或先基于数据学习离线策略与价值函数，再用于在线微调或提供参考动作。尽管各方法均表现良好，但其对样本效率的实际贡献、是否可组合及是否存在叠加效果仍不明确。本文将现有演示增强强化学习方法分为三类，通过大规模实证研究分析其优劣与组合方式，以分离各策略的贡献并识别有效的混合方案。结果表明，直接重用离线数据并采用行为克隆初始化，持续优于更复杂的离线强化学习预训练方法，对在线样本效率提升更具优势。
原文摘要 · Abstract (English)
Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。