将监督学习中的随机重排引入强化学习，提升样本效率与稳定性。
Experience Replay with Random Reshuffling
- 提出基于随机重排的体验回放方法，支持均匀与优先采样。
- 在Atari基准上验证，性能优于传统有放回采样，收敛更快。
- 适合追求训练稳定性和样本效率的深度强化学习研究者。
经验回放是强化学习中用于稳定学习和提高样本效率的关键组件，其典型实现方式是从回放缓冲区中有放回地采样过渡数据。相比之下，在具有固定数据集的监督学习中，通常每轮遍历前对数据集进行随机重排并顺序消费，称为随机重排（RR）。RR不仅具有更优的理论收敛性质，且在实践中表现优于有放回采样。为在强化学习中借鉴RR的优势，本文提出了扩展至经验回放的采样方法，涵盖均匀与优先采样场景，并通过理论分析与模拟验证其性质。我们在Atari基准上评估了这些采样方法，证明其在深度强化学习中的有效性。代码已开源：https://github.com/pfnet-research/errr。
原文摘要 · Abstract (English)
Experience replay is a key component in reinforcement learning for stabilizing learning and improving sample efficiency. Its typical implementation samples transitions with replacement from a replay buffer. In contrast, in supervised learning with a fixed dataset, it is a common practice to shuffle the dataset every epoch and consume data sequentially, which is called random reshuffling (RR). RR enjoys theoretically better convergence properties and has been shown to outperform with-replacement sampling empirically. To leverage the benefits of RR in reinforcement learning, we propose sampling methods that extend RR to experience replay, both in uniform and prioritized settings, and analyze their properties via theoretical analysis and simulations. We evaluate our sampling methods on Atari benchmarks, demonstrating their effectiveness in deep reinforcement learning. Code is available at https://github.com/pfnet-research/errr.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。