让攻击程序自动进化，突破传统提示优化的局限。
AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models
- 用代码代理动态修改攻击策略结构，支持流程重组与组件新增。
- 在11个模型上平均提升攻击成功率17.0点，前沿模型最高提升16点。
- 无需微调或标注，黑盒运行，适合安全测试与模型评估场景。
大型语言模型的自动化红队测试通常在固定的人工设计策略中优化攻击提示，而攻击策略本身保持不变。本文提出AutoRISE，通过搜索可执行的攻击程序而非单个提示来优化策略。每轮迭代中，一个编码代理编辑攻击策略，固定评估框架对生成的攻击进行评分，返回标量目标和逐例诊断，指导后续修改。该方法支持结构化变化，包括新攻击组件和控制流调整，是提示级方法无法直接表达的。我们还发布了两个在不同目标集上构建的基准套件，并在5个模型家族的11个模型上，针对7个已知越狱数据集进行评估。在未见模型上，AutoRISE相比最强基线平均提升攻击成功率17.0点，对低基线成功率的前沿目标最高提升16点。消融实验表明，这些提升源于无限制程序搜索，尤其是组合式技术与控制流修改。AutoRISE在仅推理的黑盒环境下运行，无需微调、人工标注或GPU计算。
原文摘要 · Abstract (English)
Automated red-teaming methods for large language models typically optimize attack prompts within a fixed, human-designed strategy, leaving the attack strategy itself unchanged. We instead optimize the strategy. We propose AutoRISE, a method that searches over executable attack programs rather than individual prompts. At each iteration, a coding agent edits a strategy and a fixed evaluation harness scores the resulting attacks, returning both a scalar objective and per-example diagnostics that guide subsequent edits. This allows structural changes, including new attack components and altered control flow, that prompt-level methods do not directly express. We also release two benchmark suites developed on disjoint target sets and evaluate on 11 models from five families against seven established jailbreak datasets. Across held-out models, AutoRISE improves average attack success rate by 17.0 points over the strongest baseline, and improves attack success by up to 16 points on frontier targets with low baseline success rates. Ablations against parametric and strategy-library baselines suggest that these gains arise from unrestricted program search, particularly compositional techniques and control-flow edits. AutoRISE operates in a black-box, inference-only setting, requiring no fine-tuning, human annotation, or GPU compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。