arXiv:2602.11437cs.AIcs.MA2026-02被引 1

提升多智能体强化学习在不确定环境下的鲁棒性,让每个智能体的决策更可靠。

Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorization

  • 提出分布鲁棒的个体-全局最大原则,确保各智能体决策与团队最优对齐
  • 在真实感模拟器和星际争霸环境中显著提升分布外性能,验证了鲁棒性
  • 兼容现有算法框架,无需额外奖励设计,可直接部署

合作式多智能体强化学习(MARL)通常采用集中训练、分散执行的范式,通过值函数分解方法强制满足个体-全局最大(IGM)原则,使各智能体的贪婪策略能恢复团队最优联合动作。然而,由于仿真到现实的差距、模型失配和系统噪声等环境不确定性,该方法在真实场景中的可靠性不足。本文提出分布鲁棒的IGM(DrIGM)原则,要求每个智能体的鲁棒贪婪动作与鲁棒团队最优联合动作对齐。我们证明了新定义的鲁棒个体动作值满足该原则,支持分散贪婪执行,并为整个系统提供可证明的鲁棒性保证。基于此,我们推导出符合DrIGM的鲁棒变体架构(如VDN/QMIX/QTRAN),其特点包括:(i) 使用鲁棒Q目标进行训练,(ii) 保持可扩展性,(iii) 无需定制化每智能体奖励设计,可无缝集成至现有代码库。实证结果表明,在高保真度SustainGym模拟器和星际争霸环境上,所提方法持续提升分布外性能。代码与数据已公开于https://github.com/crqu/robust-coMARL。

原文摘要 · Abstract (English)

Cooperative multi-agent reinforcement learning (MARL) commonly adopts centralized training with decentralized execution, where value-factorization methods enforce the individual-global-maximum (IGM) principle so that decentralized greedy actions recover the team-optimal joint action. However, the reliability of this recipe in real-world settings remains unreliable due to environmental uncertainties arising from the sim-to-real gap, model mismatch, and system noise. We address this gap by introducing Distributionally robust IGM (DrIGM), a principle that requires each agent's robust greedy action to align with the robust team-optimal joint action. We show that DrIGM holds for a novel definition of robust individual action values, which is compatible with decentralized greedy execution and yields a provable robustness guarantee for the whole system. Building on this foundation, we derive DrIGM-compliant robust variants of existing value-factorization architectures (e.g., VDN/QMIX/QTRAN) that (i) train on robust Q-targets, (ii) preserve scalability, and (iii) integrate seamlessly with existing codebases without bespoke per-agent reward shaping. Empirically, on high-fidelity SustainGym simulators and a StarCraft game environment, our methods consistently improve out-of-distribution performance. Code and data are available at https://github.com/crqu/robust-coMARL.

多智能体强化学习鲁棒性分布式

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。