提出联合扰动状态与动作的攻击框架,提升对多智能体强化学习模型的攻破效果。
SAJA: A State-Action Joint Attack Framework on Multi-Agent Deep Reinforcement Learning
- 分两阶段联合优化状态和动作扰动,利用网络梯度生成对抗样本。
- 在MPE环境中攻击成功率超单一扰动,且更隐蔽,防御手段无效。
- 适合研究多智能体系统安全性的研究人员,尤其关注对抗攻击者。
多智能体深度强化学习(MADRL)在自动驾驶、策略游戏等协作与竞争任务中展现出潜力,但其模型易受状态与动作上的对抗性扰动影响。现有研究仅关注状态或动作单独攻击,未充分挖掘二者协同效应。简单随机组合扰动无法实现有效协同。本文提出状态-动作联合攻击(SAJA)框架,包含两个关键阶段:(1) 状态攻击阶段,采用多步梯度上升法,结合演员网络与批评家网络计算对抗状态;(2) 动作攻击阶段,基于扰动后状态,利用批评家网络进行二次梯度上升,生成最终对抗动作。同时,在损失函数中引入启发式正则项,衡量扰动动作与原始动作的距离,增强批评家引导效果。在多智能体粒子环境(MPE)上评估表明,(1) SAJA优于且更隐蔽于单独状态或动作攻击,(2) 现有状态或动作防御方法均无法有效抵御其攻击。
原文摘要 · Abstract (English)
Multi-Agent Deep Reinforcement Learning (MADRL) has shown potential for cooperative and competitive tasks such as autonomous driving and strategic gaming. However, models trained by MADRL are vulnerable to adversarial perturbations on states and actions. Therefore, it is essential to investigate the robustness of MADRL models from an attack perspective. Existing studies focus on either state-only attacks or action-only attacks, but do not consider how to effectively joint them. Simply combining state and action perturbations such as randomly perturbing states and actions does not exploit their potential synergistic effects. In this paper, we propose the State-Action Joint Attack (SAJA) framework that has a good synergistic effects. SAJA consists of two important phases: (1) In the state attack phase, a multi-step gradient ascent method utilizes both the actor network and the critic network to compute an adversarial state, and (2) in the action attack phase, based on the perturbed state, a second gradient ascent uses the critic network to craft the final adversarial action. Additionally, a heuristic regularizer measuring the distance between the perturbed actions and the original clean ones is added into the loss function to enhance the effectiveness of the critic's guidance. We evaluate SAJA in the Multi-Agent Particle Environment (MPE), demonstrating that (1) it outperforms and is more stealthy than state-only or action-only attacks, and (2) existing state or action defense methods cannot defend its attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。