提出方法识别大规模多智能体系统中导致最差性能的脆弱智能体。
Vulnerable Agent Identification in Large-Scale Multi-Agent Reinforcement Learning
- 构建分层对抗平均场控制模型，解耦脆弱性识别与对抗策略学习。
- 通过正则化贝尔曼算子实现独立训练，降低计算复杂度。
- 可逐个识别脆弱智能体，适用于大型系统安全评估与鲁棒性分析。
当系统规模扩大时，部分智能体失效不可避免，因此识别出那些失效会导致系统性能最差的智能体至关重要。本文研究大规模多智能体强化学习（MARL）中的脆弱智能体识别（VAI）问题，将该问题建模为分层对抗去中心化平均场控制（HAD-MFC），其中上层以NP难问题形式选择脆弱智能体，下层通过平均场MARL学习其最坏情况对抗策略。两者耦合使求解困难。为此，我们首先利用Fenchel-Rockafellar对偶变换解耦层级过程，得到上层的正则化平均场贝尔曼算子，实现各层独立学习，显著降低计算复杂度；其次将上层的NP难问题重构为具有密集奖励的马尔可夫决策过程，支持通过贪心和强化学习算法逐个识别脆弱智能体，该分解可保证最优解不变。实验表明，该方法在大规模MARL及基于规则的系统中均能有效识别更多脆弱智能体，并诱导系统产生更严重的故障表现，揭示了大规模系统中每个智能体的脆弱性。代码已开源：https://github.com/Waken-dream/VAI。
原文摘要 · Abstract (English)
Partial agent failure becomes inevitable when systems scale up, making it crucial to identify the subset of agents whose failure causes worst-case system performance degradations. We study this Vulnerable Agent Identification (VAI) problem in large-scale multi-agent reinforcement learning (MARL). We frame VAI as a Hierarchical Adversarial Decentralized Mean Field Control (HAD-MFC), where the upper level selects vulnerable agents as an NP-hard task and the lower level learns their worst-case adversarial policies via mean-field MARL. The two problems are coupled together, making HAD-MFC difficult to solve. To handle this, we first decouple the hierarchical process by Fenchel-Rockafellar transform, resulting a regularized mean-field Bellman operator for upper level that enables independent learning at each level, thus reducing computational complexity. We next reformulate the upper-level NP-hard problem as an MDP with dense rewards, allowing sequential identification of vulnerable agents via greedy and RL algorithms. This decomposition provably preserves the optimal solution. Experiments show our method effectively identifies more vulnerable agents in large-scale MARL and the rule-based system, fooling system into worse failures, and reveals the vulnerability of each agent in large systems. Code available at https://github.com/Waken-dream/VAI
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。