用大模型自博弈，让智能体跳出局部最优，自动发现新策略。
Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation Models
- 用大模型生成代码和策略，跨过局部最优解
- 在追逃游戏和攻防测试中超越人工设计策略
- 适合想探索开放性创新的AI研究者
多智能体互动长期推动创新，从自然界的捕食关系到太空竞赛皆然。自博弈（SP）通过让智能体与不断进化的对手对抗，隐式构建学习路径以获得高质量策略。但传统自博弈常陷入策略单一或局部最优。本文提出基础模型自博弈（FMSP），利用大模型的代码生成能力与海量知识，突破政策空间中的局部最优。提出三类方法：(1) 基础自博弈（vFMSP）持续通过竞争优化策略；(2) 新颖性搜索自博弈（NSSP）聚焦策略多样性，不关注性能；(3) 质量-多样性自博弈（QDSP）结合两者优势，生成多样化且高质量的策略集。在连续控制场景Car Tag与简单AI安全模拟Gandalf中评估。Car Tag中，FMSP探索出多种强化学习、树搜索及启发式方法；其策略质量超过强成人设计策略。Gandalf中，FMSP可自动红队攻击大模型防御，成功突破六级逐步增强的防御体系，并能自动修补漏洞。整体上，FMSP为提升自博弈提供了新方向，开启更具创造性的开放策略发现路径。
原文摘要 · Abstract (English)
Multi-agent interactions have long fueled innovation, from natural predator-prey dynamics to the space race. Self-play (SP) algorithms try to harness these dynamics by pitting agents against ever-improving opponents, thereby creating an implicit curriculum toward learning high-quality solutions. However, SP often fails to produce diverse solutions and can get stuck in locally optimal behaviors. We introduce Foundation-Model Self-Play (FMSP), a new direction that leverages the code-generation capabilities and vast knowledge of foundation models (FMs) to overcome these challenges by leaping across local optima in policy space. We propose a family of approaches: (1) \textbf{Vanilla Foundation-Model Self-Play (vFMSP)} continually refines agent policies via competitive self-play; (2) \textbf{Novelty-Search Self-Play (NSSP)} builds a diverse population of strategies, ignoring performance; and (3) the most promising variant, \textbf{Quality-Diveristy Self-Play (QDSP)}, creates a diverse set of high-quality policies by combining the diversity of NSSP and refinement of vFMSP. We evaluate FMSPs in Car Tag, a continuous-control pursuer-evader setting, and in Gandalf, a simple AI safety simulation in which an attacker tries to jailbreak an LLM's defenses. In Car Tag, FMSPs explore a wide variety of reinforcement learning, tree search, and heuristic-based methods, to name just a few. In terms of discovered policy quality, \ouralgo and vFMSP surpass strong human-designed strategies. In Gandalf, FMSPs can successfully automatically red-team an LLM, breaking through and jailbreaking six different, progressively stronger levels of defense. Furthermore, FMSPs can automatically proceed to patch the discovered vulnerabilities. Overall, FMSPs represent a promising new research frontier of improving self-play with foundation models, opening fresh paths toward more creative and open-ended strategy discovery
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。