用进化算法发现大模型安全漏洞，生成可解释的攻击策略。
Quality-Diversity Evolution for Discovering Diverse Vulnerabilities in LLM Safety
- 基于质量-多样性进化框架，以语义层面演化攻击策略。
- 在多模型测试中发现不同漏洞模式，最高攻击成功率0.8。
- 结果可解释且可复现，适合安全研究者与模型开发者参考。
当前大模型对抗测试方法存在覆盖不足问题：人工红队难以扩展，大模型作为攻击者易出现模式坍缩，梯度方法生成不可理解乱码。本文提出一种基于语义层面的质量-多样性进化框架，利用MAP-Elites维护跨行为维度（策略类型、编码方式、提示长度）的多样化攻击档案。在GPT-4o-mini、Claude 3.5 Sonnet、Gemini 2.0 Flash及开源代码模型Devstral-small-2上的实验发现：GPT-4o-mini对假设性与多轮框架结合ROT13编码最敏感（适应度0.8），Gemini易受直接攻击与多轮结合Leetspeak影响（0.8），而Claude在所有策略下均表现模糊（最高0.4）。该方法生成可解释攻击，揭示系统性模型弱点，为提升大模型安全性提供可操作洞见，并建立未来前沿模型评估的可复现基线。代码与实验数据已开源至https://github.com/bassrehab/red-queen。
原文摘要 · Abstract (English)
Current approaches to LLM adversarial testing suffer from coverage gaps: manual red-teaming does not scale, LLM-as-attacker methods exhibit mode collapse, and gradient-based approaches produce uninterpretable gibberish. We introduce a quality-diversity evolutionary framework that operates at the semantic level, evolving interpretable attack strategies rather than token sequences. Using MAP-Elites, we maintain a diverse archive of attacks across behavioral dimensions (strategy type, encoding method, prompt length). In experiments across GPT-4o-mini, Claude 3.5 Sonnet, Gemini 2.0 Flash, and an open-weight coding model (Devstral-small-2), we discover distinct vulnerability profiles: GPT-4o-mini is vulnerable to hypothetical and multi-turn framing combined with ROT13 encoding (fitness 0.8), Gemini to direct attacks with ROT13 and multi-turn with Leetspeak (0.8), while Claude shows uniformly ambiguous responses across all strategies (max 0.4). The semantic representation produces interpretable attacks that reveal systematic, model-specific weaknesses, providing actionable insights for improving LLM safety and a reproducible baseline for evaluating future frontier models. Code and experiment artifacts are released at https://github.com/bassrehab/red-queen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。