用多智能体框架检测并阻止生成式AI的提示注入攻击
Prompt Injection Detection and Mitigation via AI Multi-Agent NLP Frameworks
- 设计多智能体系统，分工负责回复生成、输出净化和策略合规
- 在500个恶意提示测试中，注入成功率与策略违规大幅下降
- 提出四类新指标，综合评估系统对提示注入的防御能力
提示注入是生成式AI系统面临的重要挑战，可能导致非预期输出。本文提出一种面向提示注入漏洞的多智能体NLP框架，通过分层检测与强制机制进行防护。该框架协调多个专用智能体，分别负责生成响应、净化输出和确保策略合规。在500个精心设计的注入提示上评估显示，系统显著降低了注入成功率和策略违规频率。为此提出四项新指标：注入成功率（ISR）、策略覆盖频率（POF）、提示净化率（PSR）和合规一致性评分（CCS），并构建综合总注入脆弱性得分（TIVS）。系统采用OVON（Open Voice Network）框架，通过结构化JSON消息实现智能体间通信，将此前用于幻觉缓解的多智能体架构扩展至应对提示注入的独特挑战。
原文摘要 · Abstract (English)
Prompt injection constitutes a significant challenge for generative AI systems by inducing unintended outputs. We introduce a multi-agent NLP framework specifically designed to address prompt injection vulnerabilities through layered detection and enforcement mechanisms. The framework orchestrates specialized agents for generating responses, sanitizing outputs, and enforcing policy compliance. Evaluation on 500 engineered injection prompts demonstrates a marked reduction in injection success and policy breaches. Novel metrics, including Injection Success Rate (ISR), Policy Override Frequency (POF), Prompt Sanitization Rate (PSR), and Compliance Consistency Score (CCS), are proposed to derive a composite Total Injection Vulnerability Score (TIVS). The system utilizes the OVON (Open Voice Network) framework for inter-agent communication via structured JSON messages, extending a previously established multi-agent architecture from hallucination mitigation to address the unique challenges of prompt injection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。