提出新评测集与防御模型，解决提示注入防护中的误判问题
InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models
- 构建含触发词的良性样本数据集NotInject，精准测量过防御现象
- 现有模型误判率高达40%，性能下降至随机猜测水平（准确率60%）
- 新模型InjecGuard通过无偏训练策略，显著降低触发词依赖，适合安全防护研究者使用
提示注入攻击对大语言模型构成严重威胁，可导致目标劫持和数据泄露。尽管提示防护模型在防御中有效，但存在过防御问题——因触发词偏倚而将良性输入误判为恶意。为此，我们提出NotInject评测数据集，系统评估多种提示防护模型的过防御情况。该数据集包含339个富含常见提示注入触发词的良性样本，支持细粒度评估。结果显示，当前最优模型在过防御下准确率降至接近随机猜测水平（60%）。为此，我们提出InjecGuard，一种新型提示防护模型，采用新的训练策略‘免费缓解过防御’（MOF），显著减少对触发词的偏倚。InjecGuard在包括NotInject在内的多个基准上表现优异，超越现有最佳模型30.8%，提供鲁棒且开源的提示注入检测方案。代码与数据集已公开于https://github.com/leolee99/InjecGuard。
原文摘要 · Abstract (English)
Prompt injection attacks pose a critical threat to large language models (LLMs), enabling goal hijacking and data leakage. Prompt guard models, though effective in defense, suffer from over-defense -- falsely flagging benign inputs as malicious due to trigger word bias. To address this issue, we introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models. NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation. Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%). To mitigate this, we propose InjecGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words. InjecGuard demonstrates state-of-the-art performance on diverse benchmarks including NotInject, surpassing the existing best model by 30.8%, offering a robust and open-source solution for detecting prompt injection attacks. The code and datasets are released at https://github.com/leolee99/InjecGuard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。