用变异模糊测试自动挖掘攻击大模型的高成功率提示模板。
TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practice
- 基于变异模糊测试，自动生成能绕过安全限制的恶意提示模板。
- 在主流大模型上实现≥95%攻击成功率，对未见问题也有效。
- 适合安全研究人员和模型防御团队使用，开源可用。
越狱大语言模型（LLMs）旨在测试其对抗恶意提示的鲁棒性，评估模型在提示攻击下是否会产生未经授权或有害响应。本文提出TurboFuzzLLM，一种基于变异的模糊测试技术，通过黑盒用户提示接口，高效生成一组有效的越狱模板。这些模板与有害问题结合后，可诱导目标模型输出有害内容。我们指出直接应用现有模板攻击方法在实际中的局限性，并针对功能性和效率进行了改进。TurboFuzzLLM在多个公开数据集上对领先模型（包括GPT-4o与GPT-4 Turbo）实现了≥95%的攻击成功率（ASR），展现出出色的泛化能力，可应对未见过的有害问题，并有助于提升模型对提示攻击的防御能力。该工具已开源，地址为https://github.com/amazon-science/TurboFuzzLLM。
原文摘要 · Abstract (English)
Jailbreaking large-language models (LLMs) involves testing their robustness against adversarial prompts and evaluating their ability to withstand prompt attacks that could elicit unauthorized or malicious responses. In this paper, we present TurboFuzzLLM, a mutation-based fuzzing technique for efficiently finding a collection of effective jailbreaking templates that, when combined with harmful questions, can lead a target LLM to produce harmful responses through black-box access via user prompts. We describe the limitations of directly applying existing template-based attacking techniques in practice, and present functional and efficiency-focused upgrades we added to mutation-based fuzzing to generate effective jailbreaking templates automatically. TurboFuzzLLM achieves $\geq$ 95\% attack success rates (ASR) on public datasets for leading LLMs (including GPT-4o \& GPT-4 Turbo), shows impressive generalizability to unseen harmful questions, and helps in improving model defenses to prompt attacks. TurboFuzzLLM is available open source at https://github.com/amazon-science/TurboFuzzLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。