将论文中的越狱攻击转为可运行代码，实现标准化评估。
Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking
- 用多智能体系统自动把越狱论文转成可执行模块。
- 30个复现攻击平均成功率偏差仅0.26个百分点，代码复用率达82.5%。
- 适合研究模型安全、需对比不同越狱方法的学者使用。
大型语言模型的越狱技术更新速度远超基准测试，导致鲁棒性评估过时且难以跨论文比较，原因在于数据集、评测框架和判断协议的漂移。我们提出JAILBREAK FOUNDRY（JBF），通过多智能体工作流将越狱论文直接转化为可立即在统一框架中评估的可执行模块。JBF包含三个核心组件：(i) JBF-LIB 提供共享接口与可复用工具；(ii) JBF-FORGE 实现论文到模块的多智能体转换；(iii) JBF-EVAL 统一评估标准。在30个复现攻击中，JBF实现了高保真度，平均（复现-报告）攻击成功率偏差仅为+0.26个百分点。通过共享基础设施，相比原始仓库，攻击相关代码量减少超过一半，平均代码复用率达82.5%。该系统支持对所有30个攻击在10个目标模型上使用一致的GPT-4o判官进行标准化AdvBench评估。通过自动化攻击集成与标准化评估，JBF为动态安全环境提供了可持续演进的活基准解决方案。
原文摘要 · Abstract (English)
Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols. We introduce JAILBREAK FOUNDRY (JBF), a system that addresses this gap via a multi-agent workflow to translate jailbreak papers into executable modules for immediate evaluation within a unified harness. JBF features three core components: (i) JBF-LIB for shared contracts and reusable utilities; (ii) JBF-FORGE for the multi-agent paper-to-module translation; and (iii) JBF-EVAL for standardizing evaluations. Across 30 reproduced attacks, JBF achieves high fidelity with a mean (reproduced-reported) attack success rate (ASR) deviation of +0.26 percentage points. By leveraging shared infrastructure, JBF reduces attack-specific implementation code by more than half relative to original repositories and achieves an 82.5% mean reused-code ratio. This system enables a standardized AdvBench evaluation of all 30 attacks across 10 victim models using a consistent GPT-4o judge. By automating both attack integration and standardized evaluation, JBF offers a scalable solution for creating living benchmarks that keep pace with the rapidly shifting security landscape.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。