arXiv:2601.17069cs.LGcs.AI2026-01

提出无需全局信息的分布式多智能体强化学习框架,实现高效协同。

Multi-Agent Deep Reinforcement Learning Under Constrained Communications

  • 通过多跳通信的分布式图注意力网络,实现无中心化状态推断
  • 在星际争霸、足球等任务中超越主流集中训练分散执行方法
  • 适合需动态组队或环境变化的复杂协作场景

集中式训练分散执行(CTDE)是多智能体强化学习的主流范式,但其训练依赖全局状态信息,导致可扩展性差、鲁棒性弱且泛化能力不足。实际场景中,如队友增减或环境动态变化时,CTDE方法易失效且重训成本高;而分布式方法仅依赖局部信息和点对点通信,更具适应性。本文提出一种完全去中心化的多智能体强化学习框架,不依赖中央评判器或全局信息。首先设计新型分布式图注意力网络(D-GAT),通过多跳通信实现全局状态推断,各智能体以输入相关的注意力权重融合邻居特征,完全分布化完成信息整合。基于D-GAT,提出分布式图注意力MAPPO(DG-MAPPO):智能体仅使用局部观测、多跳通信和共享/平均奖励,优化本地策略与价值函数。在星露台II多智能体挑战、Google研究足球和多智能体穆乔科上的实证评估表明,该方法在多种合作任务中持续优于强基线,无论同质或异质团队均表现出更优协调能力。所提框架为鲁棒协作提供了原理清晰、可扩展的解决方案,彻底消除对集中式训练或全局可观测性的依赖。据我们所知,DG-MAPPO是首个完全摆脱特权中心信息的MARL方法,使智能体仅通过点对点通信即可学习与行动。

原文摘要 · Abstract (English)

Centralized training with decentralized execution (CTDE) has been the dominant paradigm in multi-agent reinforcement learning (MARL), but its reliance on global state information during training introduces scalability, robustness, and generalization bottlenecks. Moreover, in practical scenarios such as adding/dropping teammates or facing environment dynamics that differ from the training, CTDE methods can be brittle and costly to retrain, whereas distributed approaches allow agents to adapt using only local information and peer-to-peer communication. We present a distributed MARL framework that removes the need for centralized critics or global information. Firstly, we develop a novel Distributed Graph Attention Network (D-GAT) that performs global state inference through multi-hop communication, where agents integrate neighbor features via input-dependent attention weights in a fully distributed manner. Leveraging D-GAT, we develop the distributed graph-attention MAPPO (DG-MAPPO) -- a distributed MARL framework where agents optimize local policies and value functions using local observations, multi-hop communication, and shared/averaged rewards. Empirical evaluation on the StarCraftII Multi-Agent Challenge, Google Research Football, and Multi-Agent Mujoco demonstrates that our method consistently outperforms strong CTDE baselines, achieving superior coordination across a wide range of cooperative tasks with both homogeneous and heterogeneous teams. Our distributed MARL framework provides a principled and scalable solution for robust collaboration, eliminating the need for centralized training or global observability. To the best of our knowledge, DG-MAPPO appears to be the first to fully eliminate reliance on privileged centralized information, enabling agents to learn and act solely through peer-to-peer communication.

多智能体强化学习分布式通信约束

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。