通过优先级调度让高优先级智能体先行动，缓解多智能体强化学习中的非平稳性问题。
XP-MARL: Auxiliary Prioritization in Multi-Agent Reinforcement Learning to Address Non-Stationarity
- 按优先级顺序让智能体依次决策并传递动作，减少环境波动
- 在车联网场景中使安全性能提升84.4%，优于现有最优方法的12.8%
- 自动学习优先级策略，无需人工设定，适用于协作型多智能体系统
非平稳性是多智能体强化学习（MARL）中的根本挑战，源于各智能体同时学习并改变策略，导致每个智能体视角下的环境动态变化。本文提出开源框架XP-MARL，通过辅助优先级机制应对这一问题。该框架基于假设：优先级高的智能体先行决策并传递动作，可稳定学习过程、缓解非平稳性。其核心机制为动作传播，即高优先级智能体先行动并通信动作，为低优先级者提供更稳定的环境。此外，优先级分配不依赖预设规则，而是通过一个辅助的MARL任务学习获得，实现联合优化。在包含联网自动驾驶车辆（CAVs）的运动规划场景中，XP-MARL使基准模型的安全性提升84.4%，显著优于当前最优方法的12.8%。代码已开源。
原文摘要 · Abstract (English)
Non-stationarity poses a fundamental challenge in Multi-Agent Reinforcement Learning (MARL), arising from agents simultaneously learning and altering their policies. This creates a non-stationary environment from the perspective of each individual agent, often leading to suboptimal or even unconverged learning outcomes. We propose an open-source framework named XP-MARL, which augments MARL with auxiliary prioritization to address this challenge in cooperative settings. XP-MARL is 1) founded upon our hypothesis that prioritizing agents and letting higher-priority agents establish their actions first would stabilize the learning process and thus mitigate non-stationarity and 2) enabled by our proposed mechanism called action propagation, where higher-priority agents act first and communicate their actions, providing a more stationary environment for others. Moreover, instead of using a predefined or heuristic priority assignment, XP-MARL learns priority-assignment policies with an auxiliary MARL problem, leading to a joint learning scheme. Experiments in a motion-planning scenario involving Connected and Automated Vehicles (CAVs) demonstrate that XP-MARL improves the safety of a baseline model by 84.4% and outperforms a state-of-the-art approach, which improves the baseline by only 12.8%. Code: github.com/cas-lab-munich/sigmarl
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。