用博弈论设计抗攻击的智能体系统，让坏人也难搞垮。
MaMa: A Game-Theoretic Approach for Designing Safe Agentic Systems
- 把系统设计者和攻击者建模为博弈双方，动态优化安全策略。
- 在多个环境中，系统对抗最坏攻击时性能仍接近最优任务表现。
- 可应对更强、目标不同或使用不同大模型的攻击者，泛化能力强。
基于大语言模型的多智能体系统虽能力出众，但个别智能体失效或恶意行为会带来重大安全风险。本文研究在部分智能体被攻陷的情况下，如何自动设计仍保持安全的智能体系统。受斯塔克尔伯格安全博弈启发，将问题形式化为系统设计者（元智能体）与响应型元攻击者之间的博弈：后者选择并攻陷一组智能体以最小化安全性。提出元攻击者-元智能体（MaMa）算法，通过大模型驱动的对抗搜索实现迭代优化——元智能体提出系统设计方案，元攻击者发现最强攻击并反馈。实验证明，使用MaMa设计的系统在多种环境中均能有效抵御最坏情况下的攻击，同时性能与仅追求任务成功的系统相当。此外，所设计系统还能泛化至更强攻击者、不同攻击目标或不同底层大模型，展现出超越训练环境的鲁棒安全性。
原文摘要 · Abstract (English)
LLM-based multi-agent systems have demonstrated impressive capabilities, but they also introduce significant safety risks when individual agents fail or behave adversarially. In this work, we study the automated design of agentic systems that remain safe even when a subset of agents is compromised. Inspired by Stackelberg security games, we formalize this problem as a game between a system designer (the Meta-Agent) and a best-responding Meta-Adversary that selects and compromises a subset of agents to minimize safety. We propose Meta-Adversary-Meta-Agent (MaMa), a novel algorithm inspired by this formalization for automatically designing safe agentic systems. Our approach uses LLM-based adversarial search, where the Meta-Agent iteratively proposes system designs and receives feedback based on the strongest attacks discovered by the Meta-Adversary. Empirical evaluations across diverse environments show that systems designed with MaMa consistently defend against worst-case attacks while maintaining performance comparable to systems optimized solely for task success. Moreover, the resulting systems generalize to stronger adversaries, as well as ones with different attack objectives or underlying LLMs, demonstrating robust safety beyond the training setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。