用先验数据和策略提升强化学习采样效率，显著加速复杂任务求解。
XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policies
- 结合演示数据与预训练策略，通过增强回放缓冲区优化学习。
- 在Adroit等基准上实现当前最优性能，更新/数据比低且无需集成网络。
- 采用稳定网络结构提升分布外泛化能力，预测熵更高更鲁棒。
真实世界中的强化学习在线探索成本高昂。机器人强化学习中常引入额外数据以提高样本效率，专家示范数据对解决稀疏奖励下的困难探索任务尤为关键。尽管先验数据可用于扩充经验并预训练模型，但现有算法因未能有效利用预训练策略，未能充分发挥该设定下的潜力。本文提出XQCfD，将高效的XQC actor-critic算法扩展为可从示范数据中学习的版本，采用增强回放缓冲区、预训练策略及设计用于避免快速遗忘初始强策略的稳定策略架构。我们发现，这种稳定网络结构能生成更高熵的预测，从而在分布外情形下实现更好的策略改进。XQCfD在主流的Adroit、Robomimic和MimicGen基准上，在一系列复杂操作任务中实现了当前最优表现，尤其在低更新/数据比条件下，且无需使用集成网络。
原文摘要 · Abstract (English)
For reinforcement learning in the real world online exploration is expensive A common practice in robotic reinforcement learning is to incorporate additional data to improve sample efficiency Expert demonstration data is often crucial for solving hard exploration tasks with sparse rewards While prior data is used to augment experience and pretrain models we show that the design of existing algorithms fails to achieve the sample efficiency that is possible in this setting due to a failure to use pretrained policies effectively We propose XQCfD which extends the sample-efficient XQC actor-critic to learn from demonstrations using augmented replay buffers pretrained policies and stationary policy architectures designed to avoid rapidly unlearning the strong initial policy like prior works We show our stationary network architecture enables policy improvement out-of-distribution better than standard network architectures due to its higher entropy predictions XQCfD achieves state of the art performance across a range of complex manipulation tasks with sparse rewards from the popular Adroit Robomimic and MimicGen benchmarks -- notably with a low update-to-data ratio and no ensemble networks
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。