介绍协作多智能体强化学习中集中训练分散执行的范式与核心思想。
An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning
- 训练时利用全局信息，执行时仅依赖本地观测
- 无需运行时通信，提升系统可扩展性
- 适合协作场景，也适用于竞争或混合设置
多智能体强化学习近年来发展迅速，主流方法可分为三类：集中式训练与执行（CTE）、集中式训练分散式执行（CTDE）和分散式训练与执行（DTE）。CTDE 方法最为常见，其在训练阶段可使用全局信息（如其他智能体策略、底层状态等），而在执行阶段则仅依赖本智能体的局部观测，实现去中心化执行。该范式仅需一个独立的训练阶段，能更高效地利用可用信息，具备良好的可扩展性，无需运行时通信，通常表现优异。尽管其最自然适用于协作场景，但也可根据假设的可观测信息拓展至竞争或混合环境。本文旨在介绍协作式 MARL 中的 CTDE 设置、基本概念与常用方法，涵盖理解该领域核心思想的重要工作，未覆盖内容敬请谅解。
原文摘要 · Abstract (English)
Multi-agent reinforcement learning (MARL) has exploded in popularity in recent years. Many approaches have been developed but they can be divided into three main types: centralized training and execution (CTE), centralized training for decentralized execution (CTDE), and Decentralized training and execution (DTE). CTDE methods are the most common as they can use centralized information during training but execute in a decentralized manner -- using only information available to that agent during execution. CTDE is the only paradigm that requires a separate training phase where any available information (e.g., other agent policies, underlying states) can be used. As a result, they can be more scalable than CTE methods, do not require communication during execution, and can often perform well. CTDE fits most naturally with the cooperative case, but can be potentially applied in competitive or mixed settings depending on what information is assumed to be observed. This text is an introduction to CTDE in cooperative MARL. It is meant to explain the setting, basic concepts, and common methods. It does not cover all work in CTDE MARL as the subarea is quite extensive. I have included work that I believe is important for understanding the main concepts in the subarea and apologize to those that I have omitted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。