2022年首次系统测试大模型提示注入攻击，提出对抗微调防御方法。
Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
- 用对抗样本微调模型，提升对提示注入攻击的鲁棒性。
- 无防御时GPT-3攻击成功率31%，使用该方法后小模型攻击几乎失效。
- 发现模型越灵活越易被攻破，为后续安全机制提供基础。
本文记录了2022年针对大型语言模型中提示注入攻击的早期研究，为这一关键安全领域的演进提供了历史背景。研究聚焦于两类对抗攻击：提示注入与目标劫持，分析其构造方式，在多种大模型上进行测试并比较效果。提出并评估了一种新型防御技术——对抗微调（Adversarial Fine-Tuning）。结果表明，未采用该防御时，攻击在GPT-3系列模型上的成功率达31%；而使用该方法后，小规模GPT-3变体（Ada、Babbage、Curie）的攻击成功率降至接近零。同时发现，模型灵活性越高，越易受攻击，因此更大型的GPT-3 Davinci比GPT-2等小型模型更具脆弱性。尽管所测模型已过时，但其方法论和实证发现为现代提示注入防御研究（如指令层级系统、宪法式AI）奠定了基础。
原文摘要 · Abstract (English)
This paper documents early research conducted in 2022 on defending against prompt injection attacks in large language models, providing historical context for the evolution of this critical security domain. This research focuses on two adversarial attacks against Large Language Models (LLMs): prompt injection and goal hijacking. We examine how to construct these attacks, test them on various LLMs, and compare their effectiveness. We propose and evaluate a novel defense technique called Adversarial Fine-Tuning. Our results show that, without this defense, the attacks succeeded 31\% of the time on GPT-3 series models. When using our Adversarial Fine-Tuning approach, attack success rates were reduced to near zero for smaller GPT-3 variants (Ada, Babbage, Curie), though we note that subsequent research has revealed limitations of fine-tuning-based defenses. We also find that more flexible models exhibit greater vulnerability to these attacks. Consequently, large models such as GPT-3 Davinci are more vulnerable than smaller models like GPT-2. While the specific models tested are now superseded, the core methodology and empirical findings contributed to the foundation of modern prompt injection defense research, including instruction hierarchy systems and constitutional AI approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。