用对抗博弈提升大模型安全性和有用性，双方实时进化。
Safety Alignment of LMs via Non-cooperative Games
- 让攻击模型和防御模型在强化学习中互相对抗，持续迭代优化。
- 防御模型在保持帮助性的同时，对攻击的抵抗能力显著提升。
- 生成的攻击模型可直接用于测试其他模型的安全性，适合安全评估者使用。
确保语言模型（LM）在保持有用性的同时具备安全性，仍是人工智能对齐中的关键挑战。现有方法依赖于串行对抗训练：生成对抗性提示并微调模型以抵御它们。我们提出一种新范式：将安全对齐建模为攻击者模型与防御者模型之间的非零和博弈，两者通过在线强化学习联合训练。每个模型持续适应对方策略的演化，实现迭代改进。我们的方法采用基于成对比较的偏好奖励信号，而非单点评分，提供更鲁棒的监督，可能减少奖励黑客问题。所提出的强化学习方案AdvGame使安全与效用的帕累托前沿发生偏移，得到既更助人又更抗攻击的防御者模型。此外，生成的攻击者模型收敛为一个强大且通用的红队代理，可直接部署以探测任意目标模型。代码见github.com/facebookresearch/advgame。
原文摘要 · Abstract (English)
Ensuring the safety of language models (LMs) while maintaining their usefulness remains a critical challenge in AI alignment. Current approaches rely on sequential adversarial training: generating adversarial prompts and fine-tuning LMs to defend against them. We introduce a different paradigm: framing safety alignment as a non-zero-sum game between an Attacker LM and a Defender LM trained jointly via online reinforcement learning. Each LM continuously adapts to the other's evolving strategies, driving iterative improvement. Our method uses a preference-based reward signal derived from pairwise comparisons instead of point-wise scores, providing more robust supervision and potentially reducing reward hacking. Our RL recipe, AdvGame, shifts the Pareto frontier of safety and utility, yielding a Defender LM that is simultaneously more helpful and more resilient to adversarial attacks. In addition, the resulting Attacker LM converges into a strong, general-purpose red-teaming agent that can be directly deployed to probe arbitrary target models. Code at github.com/facebookresearch/advgame.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。