构建大规模越狱攻击数据集并提出新型评估方法，系统提升大模型安全测试能力。
The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
- 基于125个恶意种子生成11.4万条越狱提示，按14类安全攻击分类并量化策略有效性。
- 训练无需梯度搜索的生成模型，生成流畅越狱提示，规避率高达0.51，性能优于现有方法。
- 提出OPTIMUS连续评估指标，可发现隐蔽最优攻击态，突破传统二元判断局限。
越狱攻击——通过纯语言操纵绕过大模型对齐机制的对抗性提示——正构成日益严重的运行安全威胁，但该领域缺乏大规模、可复现的生成、分类与评估基础设施。本文提出三项贡献：(1) 构建包含11.4万条对抗提示的大规模组合型越狱数据集，基于JailBreakV-28K中的125个有害种子，应用912种组合策略，并通过六模型多数投票机制将其归入14类网络安全攻击（如恶意软件、钓鱼、权限提升），每种策略按类别有效性排序，实现基于具体攻击目标的策略选择；(2) 提出自动化越狱生成方法，将类别感知的LLM在Moderate和Optimal子集上指令微调，推理时仅凭有害种子即可合成流畅越狱提示，无需模板或梯度搜索，其困惑度为24–39，低于AutoDAN和AmpleGCG的40–140，对LlamaPromptGuard-2-86M的安全过滤器规避率达0.29–0.51，支持可控、可扩展的红队测试；(3) 提出OPTIMUS：一种无需训练的越狱评估器，以连续函数J(S,H)同时衡量提示与原始种子的语义相似性（S）与有害性概率（H），采用校准惩罚函数，不依赖特定任务训练，可泛化于演变中的攻击策略，揭示了二元成功率（ASR）所忽略的隐蔽最优状态（S*=0.57, H*=0.43）。11.4万条提示的实验表明，OPTIMUS可在类别层面区分弱、中、优越狱攻击，是二元评估无法提供的证据。
原文摘要 · Abstract (English)
Jailbreak attacks -- adversarial prompts that bypass LLM alignment through purely linguistic manipulation -- pose a growing operational security threat, yet the field lacks large-scale, reproducible infrastructure for generating, categorizing, and evaluating them systematically. This paper addresses that gap with three contributions. (1) Large-scale compositional jailbreak dataset. We construct 114,000 adversarial prompts by applying 912 composing strategies to 125 harmful seed prompts from JailBreakV-28K. Every prompt is assigned to one of 14 cybersecurity attack categories (e.g., malware, phishing, privilege escalation) via a six-model majority-vote pipeline, and each strategy is ranked by effectiveness per category, enabling principled strategy selection grounded in concrete adversarial objectives. (2) Automated jailbreak generation. We instruction-fine-tune category-aware LLMs on Moderate and Optimal subsets, producing models that synthesize fluent jailbreak prompts from a harmful seed at inference time -- no templates, no gradient search. Our generators achieve perplexity 24-39 versus 40-140 for AutoDAN and AmpleGCG, with safety-filter evasion rates of 0.29-0.51 Mal (LlamaPromptGuard-2-86M), enabling controllable, scalable red-teaming under realistic adversarial conditions. (3) OPTIMUS: a training-free jailbreak evaluator. OPTIMUS is a continuous metric J(S,H) that jointly captures semantic similarity between the harmful seed and the jailbreak (S) and harmfulness probability (H) via calibrated penalty functions. Unlike binary attack success rate (ASR), OPTIMUS requires no task-specific training, generalizes across evolving strategies, and exposes a stealth-optimal regime (S*=0.57, H*=0.43) that ASR misses. Experiments across 114,000 prompts confirm that OPTIMUS separates Weak, Moderate, and Optimal jailbreaks with category-level evidence binary evaluation cannot supply.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。