arXiv:2503.11726cs.LGcs.AI2025-03被引 4

提出可扩展的多智能体强化学习框架,解决状态空间爆炸与固定结构限制问题。

SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks

  • 设计无排列依赖的网络架构,支持任意数量智能体
  • 计算复杂度显著降低,训练效率提升
  • 适合动态智能体数量的协作环境,如游戏和仿真

在协作式多智能体强化学习中,智能体数量增加导致状态空间呈指数级增长,严重影响样本效率。现有方法多采用固定结构,难以适应智能体数量变化。尽管图神经网络(GNN)和自注意力机制有所进展,但其密集结构带来高计算开销。为此,本文提出新型智能体网络与非线性混合网络,确保排列等变性与可扩展性,能泛化至不同数量智能体的环境。所提网络大幅降低计算复杂度,结合可扩展超网络实现高效权重生成,并引入课程学习优化训练过程。在SMACv2与Google Research Football(GRF)上的实验表明,本方法优于现有主流方法,在多种场景下实现更优学习性能。该工作为协作式多智能体强化学习提供了更高效、灵活的框架。代码已开源:https://github.com/funny-rl/SPECTra。

原文摘要 · Abstract (English)

In cooperative multi-agent reinforcement learning (MARL), the permutation problem where the state space grows exponentially with the number of agents reduces sample efficiency. Additionally, many existing architectures struggle with scalability, relying on a fixed structure tied to a specific number of agents, limiting their applicability to environments with a variable number of entities. While approaches such as graph neural networks (GNNs) and self-attention mechanisms have progressed in addressing these challenges, they have significant limitations as dense GNNs and self-attention mechanisms incur high computational costs. To overcome these limitations, we propose a novel agent network and a non-linear mixing network that ensure permutation-equivariance and scalability, allowing them to generalize to environments with various numbers of agents. Our agent network significantly reduces computational complexity, and our scalable hypernetwork enables efficient weight generation for non-linear mixing. Additionally, we introduce curriculum learning to improve training efficiency. Experiments on SMACv2 and Google Research Football (GRF) demonstrate that our approach achieves superior learning performance compared to existing methods. By addressing both permutation-invariance and scalability in MARL, our work provides a more efficient and adaptable framework for cooperative MARL. Our code is available at https://github.com/funny-rl/SPECTra.

多智能体强化学习可扩展性图神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。