用乐观探索策略提升多智能体强化学习的最优动作采样效率
Optimistic ε-Greedy Exploration for Cooperative Multi-Agent Reinforcement Learning
- 引入乐观价值网络作为独立探索指标，增强对高回报联合动作的发现
- 在多个环境中使最终收益、胜率和收敛速度显著优于现有方法
- 适合追求高效协作与稳定收敛的多智能体系统研究者
集中训练、分散执行（CTDE）范式在合作式多智能体强化学习中广泛应用。然而，基于该范式的传统方法常因价值低估而陷入次优解。本文提出全新视角：探索阶段对最优联合动作采样不足也是关键原因。为此，我们设计了乐观ε-贪心探索策略，引入乐观行动价值网络作为解耦的探索指示器，理论上证明其以概率收敛至最大可实现回报。通过以ε概率从这些分布中采样动作，显著提高高回报联合动作的选择频率。在多种环境中的实验表明，该策略有效避免次优解，相比其他增强算法，显著提升了最终回报、胜率和收敛速度。
原文摘要 · Abstract (English)
The Centralized Training with Decentralized Execution (CTDE) paradigm is widely used in cooperative multi-agent reinforcement learning. However, conventional methods based on CTDE can suffer from value underestimation and converge to suboptimal solutions. While such underestimation is typically attributed to the representational limitations of monotonic structures, we provide a novel perspective by demonstrating that the insufficient sampling of optimal joint actions during exploration is also a critical factor. To address this problem, we propose Optimistic $ε$-Greedy Exploration. Our method introduces optimistic action-value networks that serve as decoupled exploration indicators, which we theoretically prove to converge in probability to the maximum achievable returns. By sampling actions from these distributions with a probability of $ε$, we effectively increase the selection frequency of high-return joint actions. Experimental results in various environments reveal that our strategy effectively prevents the algorithm from falling into suboptimal solutions and significantly improves final returns, win rates, and convergence speeds compared to other enhanced algorithms. Our code has been open-sourced at https://github.com/qxqxtxdy/OptimisticExploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。