用可执行代码生成攻击，让大模型更难防御
Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs
- 用多智能体自动编写和演化攻击代码，而非修改提示词
- 对强防御模型攻击成功率达85.5%，平均成功率95.9%
- 能自修复失败代码，生成的攻击更多样、更隐蔽
大型语言模型（LLMs）的自动化红队测试框架日益复杂，但多数仍局限于提示词空间优化。本文提出EvoSynth，一个自主多智能体框架，将攻击优化从提示词转向可执行代码空间。该框架不直接优化提示，而是通过多智能体系统自主设计、演化并执行基于代码的攻击算法。关键创新在于代码级自纠错机制，能根据目标模型反馈与失败尝试迭代重写攻击逻辑。实验表明，EvoSynth在对抗如Claude-Sonnet-4.5等强防御模型时达到85.5%的攻击成功率（ASR），在所有评估目标上的平均ASR为95.9%，且生成的攻击比现有方法更丰富多样。代码已开源，推动可执行代码空间中的进化合成研究。
原文摘要 · Abstract (English)
Automated red teaming frameworks for Large Language Models (LLMs) have become increasingly sophisticated, yet many still formulate attack optimization primarily in the prompt space. In other words, these methods mainly search for better attack wording or better strategy choices, but they do not search over executable code. By moving the search into code space, we can optimize not only the final attack prompt, but also the procedure that generates it, including execution flow, reusable logic, branching, and failure-driven repair. To overcome this gap, we introduce EvoSynth, an autonomous multi-agent framework that shifts the optimization space from prompts to executable code. Instead of refining prompts directly, EvoSynth employs a multi-agent system to autonomously engineer, evolve, and execute code-based attack algorithms. Crucially, it features a code-level self-correction loop, allowing it to iteratively rewrite the code-based algorithm in response to target-model feedback and failed attempts. Through extensive experiments, we demonstrate that EvoSynth achieves an 85.5\% Attack Success Rate (ASR) against highly robust models like Claude-Sonnet-4.5 and a 95.9\% average ASR across evaluated targets, while generating attacks that are significantly more diverse than those from existing methods. We release our framework to facilitate future research on evolutionary synthesis in executable code space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。