评测四大模型对人工与算法攻击的抗性，发现不同模型安全差异大。
Safeguarding Efficacy in Large Language Models: Evaluating Resistance to Human-Written and Algorithmic Adversarial Prompts
- 用1200个分层提示测试四类攻击，覆盖六种危害类型。
- Llama-2最安全（平均攻击成功率3.4%），Phi-2最易受攻击（7.0%）。
- 攻击在模型间可转移，尤其针对GPT-4最高达17%成功率。
本文系统评估了四种主流大语言模型（LLMs）在多种对抗攻击下的安全性。我们对Phi-2、Llama-2-7B-Chat、GPT-3.5-Turbo和GPT-4在四类攻击中的表现进行了测评：人工编写提示、AutoDAN、贪婪坐标梯度（GCG）以及剪枝版树状攻击（TAP）。评估基于来自SALAD-Bench数据集的1200个精心分层的提示，涵盖六类危害。结果显示模型鲁棒性差异显著，其中Llama-2整体安全最高（平均攻击成功率3.4%），而Phi-2最为脆弱（平均攻击成功率7.0%）。我们发现关键的攻击可迁移性：尽管GCG和TAP在目标模型（如Llama-2）上效果差，但在其他模型上迁移成功率显著提升，最高达17%（针对GPT-4）。统计分析（Friedman检验）显示不同危害类别间存在显著差异（p < 0.001），其中恶意使用类提示的平均攻击成功率最高（10.71%）。研究揭示跨模型安全漏洞，并为制定针对性防御策略提供依据。
原文摘要 · Abstract (English)
This paper presents a systematic security assessment of four prominent Large Language Models (LLMs) against diverse adversarial attack vectors. We evaluate Phi-2, Llama-2-7B-Chat, GPT-3.5-Turbo, and GPT-4 across four distinct attack categories: human-written prompts, AutoDAN, Greedy Coordinate Gradient (GCG), and Tree-of-Attacks-with-pruning (TAP). Our comprehensive evaluation employs 1,200 carefully stratified prompts from the SALAD-Bench dataset, spanning six harm categories. Results demonstrate significant variations in model robustness, with Llama-2 achieving the highest overall security (3.4% average attack success rate) while Phi-2 exhibits the greatest vulnerability (7.0% average attack success rate). We identify critical transferability patterns where GCG and TAP attacks, though ineffective against their target model (Llama-2), achieve substantially higher success rates when transferred to other models (up to 17% for GPT-4). Statistical analysis using Friedman tests reveals significant differences in vulnerability across harm categories ($p < 0.001$), with malicious use prompts showing the highest attack success rates (10.71% average). Our findings contribute to understanding cross-model security vulnerabilities and provide actionable insights for developing targeted defense mechanisms
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。