提出无需中心训练的分布式多智能体强化学习方法,实现去中心化协作。
Distributed Value Decomposition Networks with Networked Agents
- 通过局部估计共享目标,实现去中心化训练
- 在10个任务中逼近原有方法性能,通信损失影响小
- 适用于异构与同构智能体,适合真实物理环境
我们研究部分可观测环境下分布式训练问题,即合作型多智能体强化学习(MARL) agents 在无法获取全局信息时最大化联合累积奖励。提出分布式价值分解网络(DVDN),将联合Q函数分解为各智能体的局部Q函数。与依赖中心化训练的传统价值分解网络不同,本方法适用于无法进行中心化训练的场景,智能体可在去中心化交互中通过同伴通信学习。我们提出了两个创新算法:针对异构智能体的DVDN和针对同构智能体的DVDN(GT)。实验表明,在三个标准环境中的十个MARL任务中,尽管存在通信信息损失,两种算法仍能逼近原值分解网络的性能。
原文摘要 · Abstract (English)
We investigate the problem of distributed training under partial observability, whereby cooperative multi-agent reinforcement learning agents (MARL) maximize the expected cumulative joint reward. We propose distributed value decomposition networks (DVDN) that generate a joint Q-function that factorizes into agent-wise Q-functions. Whereas the original value decomposition networks rely on centralized training, our approach is suitable for domains where centralized training is not possible and agents must learn by interacting with the physical environment in a decentralized manner while communicating with their peers. DVDN overcomes the need for centralized training by locally estimating the shared objective. We contribute with two innovative algorithms, DVDN and DVDN (GT), for the heterogeneous and homogeneous agents settings respectively. Empirically, both algorithms approximate the performance of value decomposition networks, in spite of the information loss during communication, as demonstrated in ten MARL tasks in three standard environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。