arXiv:2605.08427cs.AIcs.GT2026-05

通过分离攻防角色提升AI安全,防止模型自我一致性失效。

The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play

论文配图:The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play
图 1 · 摘自论文原文
  • 用独立的LoRA适配器实现攻防角色分离,避免参数共享导致的自洽性崩溃
  • 相比微调效率提升100倍,在多个基准上安全性能更优
  • 适合追求高效安全训练的AI研发团队,尤其适用于大模型

自洽性红队是通过同一模型在攻防角色间进行零和博弈以提升AI安全性的方法。若自洽性收敛至纳什均衡,则模型在游戏设定下可保证安全响应。然而,因攻防共用同一模型,其参数共享机制带来理论与架构局限:可达的纳什均衡涵盖仅拒绝或类预言机式防御等非实用行为,严重限制实际应用。我们发现,当攻防共享并更新同一基础模型时,系统动态会退化为自洽性,导致攻击无法对防御者施加对抗压力。为此,我们提出锚定双策略自洽训练(Anchored Bipolicy Self-Play),在冻结的基础模型上分别训练专属攻防的LoRA适配器,保持优化稳定性的同时通过显式角色分离维持对抗压力。相较标准自洽训练,该方法参数效率最高提升100倍,且在Qwen2.5-{3B, 7B, 14B}-IT模型上跨多个主流安全基准测试中,均实现更优的安全性表现,同时不损失推理能力。跨角色对战实验进一步验证,本方法的攻防模型在对抗防御与安全性方面优于传统自洽模型。

原文摘要 · Abstract (English)

Self-play red team is an established approach to improving AI safety in which different instances of the same model play attacker and defender roles in a zero-sum game, i.e., where the attacker tries to jailbreak the defender; if self-play converges to a Nash equilibrium, the model is guaranteed to respond safely within the settings of the game. Although the parameter sharing enforced by the use of the same model for the two roles improves stability and performance, it introduces fundamental theoretical and architectural limitations. We show that the set of Nash equilibria that can be reached corresponds to a broad class of behaviours that includes trivial always refuse strategies and oracle-like defenders, thus limiting practical applicability. We then show that when attacker and defender share and update the same base model, the dynamics collapse to self-consistency, so that attacks do not enforce adversarial pressure on the defender. In response, we propose Anchored Bipolicy Self-Play, which trains distinct role-specific LoRA adapters on top of a frozen base model, thereby maintaining stable optimisation while preserving adversarial pressure through explicit role separation. In relation to standard self-play, we show up to 100x greater parameter efficiency than finetuning and consistent improvements in safety compared to self-play fine-tuned models. We evaluate on Qwen2.5-{3B, 7B,14B}-IT models across widely used safety benchmarks, showing improved robustness without loss of reasoning ability. Cross-play experiments further show that our attacker and defender models are superior to self-play in terms of adversarial defence and safety.

AI安全自洽训练攻防对抗LoRA

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。