arXiv:2502.11006cs.CRcs.AI2025-02ICML被引 3

用生成式解释辅助检测恶意提示注入,提升安全调查效率。

Prompt Inject Detection with Generative Explanation as an Investigative Tool

  • 利用大模型自动生成解释来识别恶意提示注入
  • 可帮助安全人员快速判断提示是否恶意或上下文合规
  • 适合安全研究员和平台运维人员用于高效排查风险

大型语言模型(LLMs)易受基于提示的攻击注入影响。这些注入可通过明确指令导致模型产生非预期响应,甚至突破安全限制。在安全调查中,需处理大量输入提示,其中多数为良性,而语义和上下文的主观性增加了判断难度。现有防护机制如护栏(guardrails)虽能检测部分攻击,但缺乏对检测结果的解释能力,难以支持调查人员进行优先级排序与评估。本文探索利用大模型的文本生成能力,不仅检测提示注入,还能生成可解释的推理过程,帮助安全调查者理解并判断每个检测结果的性质。该方法显著降低人工审查负担,提升调查效率。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are vulnerable to adversarial prompt based injects. These injects could jailbreak or exploit vulnerabilities within these models with explicit prompt requests leading to undesired responses. In the context of investigating prompt injects, the challenge is the sheer volume of input prompts involved that are likely to be largely benign. This investigative challenge is further complicated by the semantics and subjectivity of the input prompts involved in the LLM conversation with its user and the context of the environment to which the conversation is being carried out. Hence, the challenge for AI security investigators would be two-fold. The first is to identify adversarial prompt injects and then to assess whether the input prompt is contextually benign or adversarial. For the first step, this could be done using existing AI security solutions like guardrails to detect and protect the LLMs. Guardrails have been developed using a variety of approaches. A popular approach is to use signature based. Another popular approach to develop AI models to classify such prompts include the use of NLP based models like a language model. However, in the context of conducting an AI security investigation of prompt injects, these guardrails lack the ability to aid investigators in triaging or assessing the identified input prompts. In this applied research exploration, we explore the use of a text generation capabilities of LLM to detect prompt injects and generate explanation for its detections to aid AI security investigators in assessing and triaging of such prompt inject detections. The practical benefit of such a tool is to ease the task of conducting investigation into prompt injects.

提示注入AI安全生成解释大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。