提出InSPO算法，解决离线多智能体强化学习中的协作与分布外问题。
Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization
- 按顺序更新各智能体策略，避免选择分布外联合动作。
- 在MAgent和StarCraft II上优于现有最先进方法，收敛更快。
- 适合需要稳定协作的离线多智能体场景，如机器人协同。
离线多智能体强化学习旨在从预收集的数据集中学习最优多智能体策略。相比单智能体情形，多智能体设置涉及庞大的联合状态-动作空间及多个智能体之间的耦合行为，使离线策略优化更加复杂。本文重新审视现有离线MARL方法，发现其在某些场景下会导致不协调行为和分布外（OOD）联合动作。为此，我们提出一种新算法——样本内序列策略优化（InSPO）。InSPO以样本内方式依次更新每个智能体的策略，既避免了选择分布外联合动作，又充分考虑队友策略的更新以增强协作。此外，通过深入探索行为策略中低概率动作，InSPO有效缓解了过早收敛至次优解的问题。理论上，我们证明InSPO保证单调策略提升并收敛至量化响应均衡（QRE）。实验结果表明，该方法在MAgent和StarCraft II基准测试中显著优于当前最先进的离线MARL方法。
原文摘要 · Abstract (English)
Offline Multi-Agent Reinforcement Learning (MARL) is an emerging field that aims to learn optimal multi-agent policies from pre-collected datasets. Compared to single-agent case, multi-agent setting involves a large joint state-action space and coupled behaviors of multiple agents, which bring extra complexity to offline policy optimization. In this work, we revisit the existing offline MARL methods and show that in certain scenarios they can be problematic, leading to uncoordinated behaviors and out-of-distribution (OOD) joint actions. To address these issues, we propose a new offline MARL algorithm, named In-Sample Sequential Policy Optimization (InSPO). InSPO sequentially updates each agent's policy in an in-sample manner, which not only avoids selecting OOD joint actions but also carefully considers teammates' updated policies to enhance coordination. Additionally, by thoroughly exploring low-probability actions in the behavior policy, InSPO can well address the issue of premature convergence to sub-optimal solutions. Theoretically, we prove InSPO guarantees monotonic policy improvement and converges to quantal response equilibrium (QRE). Experimental results demonstrate the effectiveness of our method compared to current state-of-the-art offline MARL methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。