LlamaFirewall为AI代理提供实时安全防护,防范提示注入与代码漏洞。
LlamaFirewall: An open source guardrail system for building secure AI agents
- 构建三重防护:检测越狱、审计推理链、静态分析代码
- 可抵御提示注入攻击,防止代理目标偏离,拦截危险代码生成
- 支持开发者自定义规则,适配不同场景的安全需求
大语言模型已演变为能执行复杂任务的自主智能体,如编辑生产代码、编排工作流,甚至基于不可信输入(如网页、邮件)采取高风险操作。这些能力带来新型安全威胁,现有措施如微调或聊天机器人式防护无法充分应对。由于风险高且缺乏确定性解决方案,亟需实时监控的最终防御层,并支持系统级、场景特定的安全策略定义与执行。我们提出LlamaFirewall,一个开源安全导向的防护框架,作为对抗AI代理安全风险的最终防线。该框架通过三项核心防护机制实现风险缓解:PromptGuard 2——通用越狱检测器,性能达到当前最优;Agent Alignment Checks——链式思考审计器,检查代理推理过程中的提示注入与目标偏差,虽仍处实验阶段,但在一般场景下对间接注入的防护效果优于已有方法;CodeShield——高效可扩展的在线静态分析引擎,用于阻止编码代理生成不安全或危险代码。此外,还提供易用的可定制扫描器,使任何能编写正则表达式或LLM提示的开发者都能快速更新代理安全规则。
原文摘要 · Abstract (English)
Large language models (LLMs) have evolved from simple chatbots into autonomous agents capable of performing complex tasks such as editing production code, orchestrating workflows, and taking higher-stakes actions based on untrusted inputs like webpages and emails. These capabilities introduce new security risks that existing security measures, such as model fine-tuning or chatbot-focused guardrails, do not fully address. Given the higher stakes and the absence of deterministic solutions to mitigate these risks, there is a critical need for a real-time guardrail monitor to serve as a final layer of defense, and support system level, use case specific safety policy definition and enforcement. We introduce LlamaFirewall, an open-source security focused guardrail framework designed to serve as a final layer of defense against security risks associated with AI Agents. Our framework mitigates risks such as prompt injection, agent misalignment, and insecure code risks through three powerful guardrails: PromptGuard 2, a universal jailbreak detector that demonstrates clear state of the art performance; Agent Alignment Checks, a chain-of-thought auditor that inspects agent reasoning for prompt injection and goal misalignment, which, while still experimental, shows stronger efficacy at preventing indirect injections in general scenarios than previously proposed approaches; and CodeShield, an online static analysis engine that is both fast and extensible, aimed at preventing the generation of insecure or dangerous code by coding agents. Additionally, we include easy-to-use customizable scanners that make it possible for any developer who can write a regular expression or an LLM prompt to quickly update an agent's security guardrails.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。