arXiv:2510.00274cs.AIcs.LG2025-10被引 1

让多智能体强化学习决策过程可解释,提升安全性和学习效率。

MAGIC-MASK: Multi-Agent Guided Inter-Agent Collaboration with Mask-Based Explainability for Reinforcement Learning

  • 通过掩码共享与协作机制,实现多智能体间关键状态的协同识别。
  • 在高速公路和足球环境测试中,解释精度更高,学习速度提升30%以上。
  • 适合需要透明决策的自动驾驶、机器人协作等安全敏感场景。

理解深度强化学习智能体的决策过程,仍是其在安全关键及多智能体环境中部署的主要挑战。现有可解释方法如StateMask虽能识别关键状态,但受限于计算成本、探索覆盖率不足,且难以适应多智能体设置。为此,我们提出数学基础扎实的MAGIC-MASK框架,将基于扰动的可解释性扩展至多智能体强化学习。该方法融合近端策略优化(PPO)、自适应epsilon-greedy探索与轻量级跨智能体协作,共享掩码状态信息与同伴经验。各智能体通过显著性引导掩码并传递基于奖励的洞察,显著缩短关键状态发现时间,提升解释保真度,实现更快更稳健的学习。核心创新在于通过轨迹扰动、奖励保真度分析与KL散度正则化,构建统一数学形式,将可解释性从单智能体推广至多智能体系统。实验在单/多智能体基准上验证,包括多智能体高速路驾驶与Google Research Football环境,结果表明MAGIC-MASK在解释保真度、学习效率与策略鲁棒性方面持续优于现有先进基线,同时提供可解释且可迁移的推理过程。

原文摘要 · Abstract (English)

Understanding the decision-making process of Deep Reinforcement Learning agents remains a key challenge for deploying these systems in safety-critical and multi-agent environments. While prior explainability methods like StateMask, have advanced the identification of critical states, they remain limited by computational cost, exploration coverage, and lack of adaptation to multi-agent settings. To overcome these limitations, we propose a mathematically grounded framework, MAGIC-MASK (Multi-Agent Guided Inter-agent Collaboration with Mask-Based Explainability for Reinforcement Learning), that extends perturbation-based explanation to Multi-Agent Reinforcement Learning. Our method integrates Proximal Policy Optimization, adaptive epsilon-greedy exploration, and lightweight inter-agent collaboration to share masked state information and peer experience. This collaboration enables each agent to perform saliency-guided masking and share reward-based insights with peers, reducing the time required for critical state discovery, improving explanation fidelity, and leading to faster and more robust learning. The core novelty of our approach lies in generalizing explainability from single-agent to multi-agent systems through a unified mathematical formalism built on trajectory perturbation, reward fidelity analysis, and Kullback-Leibler divergence regularization. This framework yields localized, interpretable explanations grounded in probabilistic modeling and multi-agent Markov decision processes. We validate our framework on both single-agent and multi-agent benchmarks, including a multi-agent highway driving environment and Google Research Football, demonstrating that MAGIC-MASK consistently outperforms state-of-the-art baselines in fidelity, learning efficiency, and policy robustness while offering interpretable and transferable explanations.

强化学习可解释性多智能体掩码机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。