构建模块化评测框架,公平比较多轮越狱攻击的真正效果。
MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks

- 将攻击拆分为五个可独立评估的模块,实现公平对比。
- 发现预算和评估函数是影响结果的关键混淆因素。
- 组件级分析表明提示生成最关键,随机采样策略已足够强。
多轮越狱攻击利用大语言模型对对话上下文的累积理解,通过渐进式引导而非直接请求来获取不当回答。现有方法常在不同预算、评估者、重试规则和策略生成方式下进行黑箱测试,导致性能差异难以归因于攻击机制本身还是实验条件。为此,我们提出MT-JailBench,一个模块化评测框架,在固定条件下评估多轮越狱攻击。该框架将攻击分解为评估函数、攻击策略、提示生成、提示优化和流程控制五个交互模块,支持组件级分析与公平比较。实验发现,控制轮次、重试次数、交互数、策略采样数和评估者后,攻击排名显著变化;提示生成是性能差异的主要来源,优化与流程控制贡献较小;且显式动态策略生成并非必需,固定策略的随机采样即可达到媲美复杂机制的效果。重组最优组件后,得到的攻击配置超越原始方法,并在多种目标LLM上具有泛化能力。因此,MT-JailBench为多轮越狱攻击的比较、组件影响分析及更强红队测试提供了可复用框架。
原文摘要 · Abstract (English)
Multi-turn jailbreaks exploit the ability of large language models to accumulate and act on conversational context. Instead of stating a harmful request directly, an attacker can gradually steer the conversation toward an unsafe answer. Recent methods demonstrate this risk, but they are usually evaluated as black-box pipelines with different budgets, judges, retry rules, and strategy generation procedures. As a result, it is often unclear whether reported gains reflect stronger attack mechanisms or different experimental conditions. We introduce MT-JailBench, a modular evaluation framework for benchmarking multi-turn jailbreaks under fixed conditions. MT-JailBench implements each attack as five interacting modules: evaluation function, attack strategy, prompt generation, prompt refinement, and flow control. This design enables fair comparison across attack methods and component-wise analysis of what drives attack success. Using MT-JailBench, we find that resource budgets and evaluation functions are major confounders: controlling turns, retries, interactions, sampled strategies, and judges substantially change the ranking of attacks. At the component level, prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. We also find that explicit dynamic strategy generation is not always necessary; stochastic sampling from a fixed strategy can rival more elaborate diversification mechanisms. Finally, recomposing the best components yields a strong attack configuration that outperforms its source attacks and generalizes across diverse target LLMs. MT-JailBench therefore provides a modular framework for comparing multi-turn jailbreaks, understanding the impact of components, and guiding stronger red-teaming evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。