用在线自对弈强化学习让AI自己攻防，持续提升语言模型安全性。
Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models
- 单一策略交替扮演攻击者与防御者，实时生成并应对对抗性提示。
- 在14个基准上使安全性能提升最高达95%，攻击多样性增加17.80%。
- 适用于追求主动防御的AI安全研究者，推动从被动修补到自我进化。
传统大语言模型安全对齐采用反应式、断续的流程：攻击者利用静态模型漏洞，防御者随后修补。这种顺序机制导致攻击者过拟合旧有漏洞，而防御者始终落后于新威胁。为此，我们提出Self-RedTeam，首个完全在线的自对弈多智能体强化学习（MARL）算法，实现攻击者与防御者的持续共演化。单一策略同时扮演攻防角色，生成对抗性提示并加以防御，由奖励模型裁定结果。每种角色使用隐藏的思维链进行战略规划。基于双人零和博弈理论，我们建立理论安全保证：若博弈收敛至纳什均衡，防御者可对任意对抗输入生成安全响应。实证表明，Self-RedTeam在五个来自Llama和Qwen系列的模型上具有泛化能力，攻击多样性提升17.80%（以SBERT衡量），并使基于RLHF训练的模型在14个基准上的安全性最高提升95%。本工作推动安全对齐从被动修补转向主动共演化，实现大语言模型通过在线自对弈MARL的自我改进。
原文摘要 · Abstract (English)
Conventional large language model (LLM) safety alignment relies on a reactive, disjoint loop: attackers exploit a static model, then defenders patch exposed vulnerabilities. This sequential setup leads to attackers overfitting obsolete exploits while defenders perpetually lag behind emerging threats. To address this, we introduce Self-RedTeam, the first fully online self-play multi-agent reinforcement learning (MARL) algorithm that continuously co-evolves attacker and defender for robust safety alignment. A single policy self-plays as both attacker and defender, generating adversarial prompts and defending against them, with a reward model adjudicating outcomes. Each role uses hidden chain-of-thought for strategic planning. Grounded in two-player zero-sum game theory, we establish a theoretical safety guarantee: if the game converges to Nash Equilibrium, the defender produces safe responses against any adversarial input. Empirically, Self-RedTeam generalizes across five models from the Llama and Qwen families, uncovering more diverse attacks (+17.80% SBERT) and improving safety of RLHF-trained models by up to 95% across 14 benchmarks. Our work motivates a shift from reactive patching to proactive co-evolution, enabling LLM safety self-improvement via online self-play MARL. Link to code: https://github.com/mickelliu/selfplay-redteaming
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。