提出分组训练+去中心化执行框架,大幅提升大规模多智能体系统性能。
GTDE: Grouped Training with Decentralized Execution for Multi-agent Actor-Critic
- 按观察历史自适应分组,用Gumbel-Sigmoid实现可微分采样
- 495智能体环境下总奖励提升382%,64智能体下胜率100%
- 无需中心模块,适合大规模多智能体协同与对抗场景
多智能体强化学习的快速发展催生了多种策略训练范式。去中心化训练与执行(DTDE)和集中式训练去中心化执行(CTDE)虽广泛应用,但随着智能体数量增加,其性能指标(如胜率、总奖励)显著下降。为此,本文提出分组训练去中心化执行(GTDE)新范式,无需中心模块,仅依赖本地信息,有效满足大规模多智能体系统的训练需求。首先引入自适应分组模块,根据智能体观察历史划分不同组别;为支持端到端训练,采用Gumbel-Sigmoid实现分组分布上的高效点对点采样并保障梯度回传。针对组内成员数不确定的问题,设计两种方法实现组信息聚合。实验表明,在495智能体的合作环境中,GTDE平均总奖励提升382%;在64智能体的对抗环境中,对基准方法达到100%胜率。
原文摘要 · Abstract (English)
The rapid advancement of multi-agent reinforcement learning (MARL) has given rise to diverse training paradigms to learn the policies of each agent in the multi-agent system. The paradigms of decentralized training and execution (DTDE) and centralized training with decentralized execution (CTDE) have been proposed and widely applied. However, as the number of agents increases, the inherent limitations of these frameworks significantly degrade the performance metrics, such as win rate, total reward, etc. To reduce the influence of the increasing number of agents on the performance metrics, we propose a novel training paradigm of grouped training decentralized execution (GTDE). This framework eliminates the need for a centralized module and relies solely on local information, effectively meeting the training requirements of large-scale multi-agent systems. Specifically, we first introduce an adaptive grouping module, which divides each agent into different groups based on their observation history. To implement end-to-end training, GTDE uses Gumbel-Sigmoid for efficient point-to-point sampling on the grouping distribution while ensuring gradient backpropagation. To adapt to the uncertainty in the number of members in a group, two methods are used to implement a group information aggregation module that merges member information within the group. Empirical results show that in a cooperative environment with 495 agents, GTDE increased the total reward by an average of 382\% compared to the baseline. In a competitive environment with 64 agents, GTDE achieved a 100\% win rate against the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。