用博弈论训练模型识别自适应提示注入攻击
DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks
- 构建对抗性场景，让模型在博弈中学会检测隐蔽攻击
- 在多个数据集和模型上均有效检测现有及自适应攻击
- 适合安全研究者与大模型应用开发者参考
集成大语言模型的应用和代理容易受到提示注入攻击，攻击者通过向输入中插入恶意提示以诱导期望输出。现有检测方法对前沿攻击，尤其是自适应攻击，效果有限。本文提出DataSentinel，一种基于博弈论的检测方法，通过微调大模型来识别被策略性规避检测的注入提示。将该问题建模为极小极大优化问题，目标是使模型能检测强自适应攻击。进一步提出基于梯度的方法，交替求解内层最大化与外层最小化问题。在多个基准数据集和大模型上的评估表明，DataSentinel能有效检测现有及自适应提示注入攻击。
原文摘要 · Abstract (English)
LLM-integrated applications and agents are vulnerable to prompt injection attacks, where an attacker injects prompts into their inputs to induce attacker-desired outputs. A detection method aims to determine whether a given input is contaminated by an injected prompt. However, existing detection methods have limited effectiveness against state-of-the-art attacks, let alone adaptive ones. In this work, we propose DataSentinel, a game-theoretic method to detect prompt injection attacks. Specifically, DataSentinel fine-tunes an LLM to detect inputs contaminated with injected prompts that are strategically adapted to evade detection. We formulate this as a minimax optimization problem, with the objective of fine-tuning the LLM to detect strong adaptive attacks. Furthermore, we propose a gradient-based method to solve the minimax optimization problem by alternating between the inner max and outer min problems. Our evaluation results on multiple benchmark datasets and LLMs show that DataSentinel effectively detects both existing and adaptive prompt injection attacks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。