arXiv:2506.07548cs.AIcs.RO2025-06被引 1

动态调整对手强度,让多智能体强化学习突破固定难度瓶颈。

Overcoming Environmental Meta-Stationarity in MARL via Adaptive Curriculum and Counterfactual Group Advantage

  • 在线根据胜率自适应调整对手强度,避免学习陷入局部最优。
  • 在SMAC超难地图上平均胜率40%,回报达17.85,优于多个基线方法。
  • 适合需要强泛化能力的复杂协作任务,如游戏对抗与机器人协同。

多智能体强化学习(MARL)在应对脚本化对手的合作任务中已达到竞争性表现,但多数方法在整个训练过程中保持单一固定难度。我们称此为环境元平稳性,发现它限制了策略泛化并引导学习趋向浅层局部最优。为打破该困境,我们提出CL-MARL,一种动态课程学习框架,通过胜率信号在线自适应调整对手强度,随代理掌握任务而推进或回退任务难度。其调度器FlexDiff融合动量趋势估计与滑动窗口双曲线监控训练与评估回报,实现无需人工调参的稳定难度过渡。由于动态课程加剧非平稳性并稀疏全局奖励,我们引入反事实组相对策略优势(CGRPA),在GRPO风格组相对优化基础上加入反事实基线,以解耦团队动态变化下的个体贡献。在星际争霸多智能体挑战(SMAC)上,CL-MARL在超难地图上取得40%的平均胜率,平均回合回报为17.85,较QMIX、OW-QMIX、DER、EMC和MARR等基线平均提升2.94;在8m_vs_9m和3s5z_vs_3s6z地图上,其峰值胜率达成速度分别快1.28倍和1.42倍于最强基线。代码已公开于https://github.com/NICE-HKU/CL2MARL-SMAC。

原文摘要 · Abstract (English)

Multi-agent reinforcement learning (MARL) has reached competitive performance on cooperative tasks against scripted adversaries, yet most methods train agents at a single fixed difficulty throughout the entire run. We term this static-difficulty regime environmental meta-stationarity and show that it caps policy generalization and steers learning toward shallow local optima. To break this regime, we propose CL-MARL, a dynamic curriculum learning framework that adapts opponent strength online from win-rate signals, advancing or regressing the task as agents master it. Its scheduler, FlexDiff, fuses momentum-based trend estimation with sliding-window dual-curve monitoring of training and evaluation returns, yielding stable difficulty transitions without manual tuning. Because a moving curriculum amplifies non-stationarity and sparsifies global rewards, we introduce the Counterfactual Group Relative Policy Advantage (CGRPA), which extends GRPO-style group-relative optimization with counterfactual baselines to disentangle each agent's contribution under shifting team dynamics. On the StarCraft Multi-Agent Challenge (SMAC), CL-MARL attains a 40% mean win rate on the super-hard maps with an average episode return of 17.85, exceeding the QMIX, OW-QMIX, DER, EMC, and MARR baselines by +2.94 on average, while reaching its peak win rate roughly 1.28faster on 8m_vs_9m and 1.42 faster on 3s5z_vs_3s6z than the strongest baseline. The implementation is publicly available at https://github.com/NICE-HKU/CL2MARL-SMAC.

多智能体强化学习动态课程SMAC

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。