用有向无环图组织智能体,实现灵活协作的多智能体强化学习。
Reinforcement Networks: novel framework for collaborative Multi-Agent Reinforcement Learning tasks
- 将智能体构造成有向无环图,支持任意结构的层级协作。
- 在多个任务上超越传统MARL基线,提升协同性能。
- 适合构建复杂、可扩展的多智能体系统,研究者可复现验证。
现代人工智能系统通常包含多个可学习组件,可自然地组织为图结构。核心挑战在于无需受限架构或训练假设即可实现端到端训练。这类任务契合协作式多智能体强化学习(MARL)领域。本文提出强化网络(Reinforcement Networks)框架,将智能体作为有向无环图(DAG)的顶点组织。该结构将层级强化学习拓展至任意DAG,实现灵活的信用分配与可扩展协调,同时避免对严格拓扑结构、完全集中训练等限制。我们形式化了该框架的训练与推理方法,并与LevelEnv概念结合,支持可复现的构建、训练与评估。在多个协作MARL任务中,我们构建的强化网络模型均优于标准基线。除了实证提升,该框架统一了层级、模块化与图结构视角,为设计和训练复杂多智能体系统提供了原则性路径。最后,我们提出理论与实践方向:更丰富的图形态、组合式课程学习、图感知探索,使强化网络成为可扩展、结构化MARL研究的新基础。
原文摘要 · Abstract (English)
Modern AI systems often comprise multiple learnable components that can be naturally organized as graphs. A central challenge is the end-to-end training of such systems without restrictive architectural or training assumptions. Such tasks fit the theory and approaches of the collaborative Multi-Agent Reinforcement Learning (MARL) field. We introduce Reinforcement Networks, a general framework for MARL that organizes agents as vertices in a directed acyclic graph (DAG). This structure extends hierarchical RL to arbitrary DAGs, enabling flexible credit assignment and scalable coordination while avoiding strict topologies, fully centralized training, and other limitations of current approaches. We formalize training and inference methods for the Reinforcement Networks framework and connect it to the LevelEnv concept to support reproducible construction, training, and evaluation. We demonstrate the effectiveness of our approach on several collaborative MARL setups by developing several Reinforcement Networks models that achieve improved performance over standard MARL baselines. Beyond empirical gains, Reinforcement Networks unify hierarchical, modular, and graph-structured views of MARL, opening a principled path toward designing and training complex multi-agent systems. We conclude with theoretical and practical directions - richer graph morphologies, compositional curricula, and graph-aware exploration. That positions Reinforcement Networks as a foundation for a new line of research in scalable, structured MARL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。