NGU算法拓展至多智能体环境，共享经验提升学习稳定性。
Extending NGU to Multi-Agent RL: A Preliminary Study
- 用共享经验回放缓冲区增强多智能体探索能力
- 相比基线，回报更高且学习更稳定，尤其在共享缓冲区下
- 适合需要稳定探索的多智能体强化学习场景
NGU算法在稀疏奖励任务中表现优异，通过结合周期性新颖性和内在动机实现。本文将其扩展至多智能体环境，在PettingZoo中的simple_tag环境中评估。相较于多智能体DQN基线，NGU实现适度更高的回报和更稳定的训练过程。研究了三项设计选择：(1) 共享与独立经验回放缓冲区，(2) 使用不同k阈值共享周期性新颖性，(3) 采用异构beta参数。结果表明，共享回放缓冲区性能最佳，凸显经验共享与内在探索结合的重要性。当k=1时，新颖性共享效果相当，但k较大时会损害学习；异构beta值未优于小而统一的值。这些发现表明，当经验共享且内在探索信号合理调优时，NGU可有效应用于多智能体设置。
原文摘要 · Abstract (English)
The Never Give Up (NGU) algorithm has proven effective in reinforcement learning tasks with sparse rewards by combining episodic novelty and intrinsic motivation. In this work, we extend NGU to multi-agent environments and evaluate its performance in the simple_tag environment from the PettingZoo suite. Compared to a multi-agent DQN baseline, NGU achieves moderately higher returns and more stable learning dynamics. We investigate three design choices: (1) shared replay buffer versus individual replay buffers, (2) sharing episodic novelty among agents using different k thresholds, and (3) using heterogeneous values of the beta parameter. Our results show that NGU with a shared replay buffer yields the best performance and stability, highlighting that the gains come from combining NGU intrinsic exploration with experience sharing. Novelty sharing performs comparably when k = 1 but degrades learning for larger values. Finally, heterogeneous beta values do not improve over a small common value. These findings suggest that NGU can be effectively applied in multi-agent settings when experiences are shared and intrinsic exploration signals are carefully tuned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。