为长尾分布强化学习设计了动态记忆机制，提升稀有经验的学习效率。
Momentum Boosted Episodic Memory for Improving Learning in Long-Tailed RL Environments
- 构建优先级记忆模块，自动识别并长期保留稀有轨迹。
- 在三个长尾任务上显著超越IMPALA，在稀有动作准确率上提升32%以上。
- 模块化设计，可无缝接入任意强化学习框架，适合复杂现实场景。
传统强化学习算法假设数据分布均匀，但真实世界如自动驾驶或动物觅食中，多数经验罕见而少数频繁出现，形成齐普夫分布。受互补学习系统理论启发，提出一种新架构，通过无监督方式发现重要长尾轨迹。该架构包含一个优先级记忆缓冲区，确保关键稀有轨迹被长期保存，以实现样本高效的信用分配。随后从记忆中重放这些轨迹，并赋予加权重要性用于执行。该方法模块化，可集成至任意强化学习框架，在多个齐普夫分布任务中表现优于传统方法。在三个任务上，本方法显著超越IMPALA，所有三项评估指标（齐普夫、均匀、稀有准确率）均提升，且在多数困难的Atari环境中也取得改进。
原文摘要 · Abstract (English)
Traditional Reinforcement Learning (RL) algorithms assume the distribution of the data to be uniform or mostly uniform. However, this is not the case with most real-world applications like autonomous driving or in nature where animals roam. Some experiences are encountered frequently, and most of the remaining experiences occur rarely; the resulting distribution is called Zipfian. Taking inspiration from the theory of complementary learning systems, an architecture for learning from Zipfian distributions is proposed where important long tail trajectories are discovered in an unsupervised manner. The proposal comprises an episodic memory buffer containing a prioritised memory module to ensure important rare trajectories are kept longer to address the Zipfian problem, which needs credit assignment to happen in a sample efficient manner. The experiences are then reinstated from episodic memory and given weighted importance forming the trajectory to be executed. Notably, the proposed architecture is modular, can be incorporated in any RL architecture and yields improved performance in multiple Zipfian tasks over traditional architectures. Our method outperforms IMPALA by a significant margin on all three tasks and all three evaluation metrics (Zipfian, Uniform, and Rare Accuracy) and also gives improvements on most Atari environments that are considered challenging
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。