用数学语言构建可验证的AI代理安全系统,防止恶意指令入侵。
The LLMbda Calculus: AI Agents, Conversations, and Information Flow
- 引入带溯源标签的λ演算,让信息流控制成为程序自身可表达的逻辑。
- 在银行代理测试中,安全模式下表现接近顶尖系统,且抵御99%攻击。
- 代码与证明均用形式化工具验证,是首个可机器验证安全性的代理框架。
大型语言模型正越来越多地作为智能体部署:它们规划、调用工具、读取不可信数据并执行结果。这使它们面临提示注入攻击:本应仅被读取的数据被当作指令执行。最可靠的防御方法是用溯源替代内容检查:按来源分类数据,通过职责分离(双模型模式)和信息流控制区分可信与不可信数据。然而现有系统难以完全信赖:流追踪易出错,有意放松策略难审计,双模型架构又固化于设计中。本文提出LLMbda,一种无类型、传值求值的λ演算,使基于溯源的防御既可表达又可形式化证明安全,且不绑定特定架构。它将智能体系统的操作核心作为一等构造:可分叉与清除的对话、代码生成,以及每个值携带标签并由每步归约传递的动态信息流控制。隔离变为程序可声明的策略,重新分类成为显式且可审计的构造。核心成果是在整个演算上建立终止无关的概率非干扰定理,包含代码生成型智能体;另有绝缘变体,在攻击者任意选择所有不可信输入时仍成立。经验证的解释器本身即为调用模型的框架,据我们所知,这是首个其可执行代码受机器检查安全定理保护的LLM智能体框架,因此每个智能体都继承该保证。在AgentDojo银行基准测试中,基于LLMbda构建的智能体,安全机制始终开启,其性能匹配领先双模型防御方案CaMeL(但关闭策略检查后性能减半),并抵御了1296次攻击中的1294次。框架与全部证明均在Lean中实现。
原文摘要 · Abstract (English)
Large language models are increasingly deployed as agents: they plan, call tools, read untrusted data, and act on the results. This exposes them to prompt injection: data meant only to be read is obeyed as an instruction. The most principled defences replace content inspection with provenance: classifying data by source and keeping trusted and untrusted apart through a separation of duty (the dual-LLM pattern) and information-flow control. Yet the leading systems are hard to fully trust: flow tracking is easy to get wrong, deliberate relaxations are hard to audit, and the dual-LLM pattern is hard-wired into the architecture. We present LLMbda, an untyped call-by-value lambda calculus that makes provenance-based defence both expressible and provably sound, without committing to an architecture. It adds the operational core of agentic systems as first-class constructs: prompt-response conversations that can be forked and cleared, code generation, and dynamic information-flow control in which every value carries a label that every reduction propagates. Isolation becomes a policy a program expresses, and reclassification an explicit, auditable construct. Our central result is a termination-insensitive probabilistic noninterference theorem over the whole calculus, including code-generating agents, with an insulated variant that holds even when the attacker chooses all untrusted inputs. The verified interpreter is itself the harness that calls the model, to our knowledge the first LLM agent harness whose executable is the subject of machine-checked security theorems, so every agent inherits the guarantee. On the AgentDojo banking benchmark, an agent built within LLMbda, enforcement always on, matches the utility of CaMeL, a leading dual-LLM defence, run without its policy checks (which halve its utility), and resists all but two of 1296 attacked runs. Our harness and all proofs are in Lean.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。