提出递归更新策略的K级梯度方法，解决多智能体强化学习中的协作错配问题。
$K$-Level Policy Gradients for Multi-Agent Reinforcement Learning
- 递归地考虑其他智能体策略的实时更新，实现更精准的策略优化
- 在星际争霸和多智能体MuJoCo中性能超越现有算法
- 理论证明有限迭代下可单调收敛至局部纳什均衡，适合复杂协作场景
深度多智能体强化学习（MARL）中的演员-评论家算法通常基于当前其他智能体的策略进行策略更新。这种方法虽直接，但未考虑同一更新步骤中其他智能体的策略变化，导致协作失调。本文提出K级策略梯度（KPG），通过递归地针对其他智能体的更新后策略进行优化，加速发现有效协作策略。理论上证明，在特定条件下，有限迭代的KPG可单调收敛至局部纳什均衡。我们通过将KPG应用于MAPPO、MADDPG和FACMAC等深度MARL算法，提供了合理实现方案。实验表明，KPG在星际争霸II和多智能体MuJoCo任务中显著优于现有深度MARL算法。
原文摘要 · Abstract (English)
Actor-critic algorithms for deep multi-agent reinforcement learning (MARL) typically employ a policy update that responds to the current strategies of other agents. While being straightforward, this approach does not account for the updates of other agents at the same update step, resulting in miscoordination. In this paper, we introduce the $K$-Level Policy Gradient (KPG), a method that recursively updates each agent against the updated policies of other agents, speeding up the discovery of effective coordinated policies. We theoretically prove that KPG with finite iterates achieves monotonic convergence to a local Nash equilibrium under certain conditions. We provide principled implementations of KPG by applying it to the deep MARL algorithms MAPPO, MADDPG, and FACMAC. Empirically, we demonstrate superior performance over existing deep MARL algorithms in StarCraft II and multi-agent MuJoCo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。