为大模型智能体安全提供上下文化定义框架,解决误判与漏洞并存难题。
A Framework for Formalizing LLM Agent Security
- 从任务对齐、动作对齐等四方面定义智能体安全的上下文特性
- 将提示注入等攻击统一为对安全属性的违反,定位更精准
- 适用于安全研究人员和防御系统设计者,推动可验证安全机制发展
大模型智能体的安全性具有强上下文依赖性:同一行为可能因指令来源、目标意图或是否达成目标而被判定为合法或违规。现有安全定义常忽略此特性,导致防御措施面临效用-安全权衡:全场景通用防御降低效率,局部防御则留下漏洞。本文提出一套上下文安全框架,包含四项核心安全属性:任务对齐(执行授权目标)、动作对齐(行为服务于目标)、源授权(仅响应认证来源)、数据隔离(信息流不越权)。引入一组预言机函数,用于在执行过程中验证这些属性是否被违反。基于该框架,我们重新形式化了间接提示注入、直接提示注入、越狱、任务漂移和内存污染等典型攻击,将其定义为对一项或多项安全属性的违背;同时将防御机制视为增强预言机或执行属性检查的手段。最后,讨论了由该框架开启的若干重要研究方向。
原文摘要 · Abstract (English)
Security in LLM agents is inherently contextual. For example, the same action taken by an agent may represent legitimate behavior or a security violation depending on whose instruction led to the action, what objective is being pursued, and whether the action serves that objective. However, existing definitions of security attacks against LLM agents often fail to capture this contextual nature. As a result, defenses face a fundamental utility-security tradeoff: applying defenses uniformly across all contexts can lead to significant utility loss, while applying defenses in insufficient or inappropriate contexts can result in security vulnerabilities. In this work, we present a framework that systematizes existing attacks and defenses from the perspective of contextual security. To this end, we propose four security properties that capture contextual security for LLM agents: task alignment (pursuing authorized objectives), action alignment (individual actions serving those objectives), source authorization (executing commands from authenticated sources), and data isolation (ensuring information flows respect privilege boundaries). We further introduce a set of oracle functions that enable verification of whether these security properties are violated as an agent executes a user task. Using this framework, we reformalize existing attacks, such as indirect prompt injection, direct prompt injection, jailbreak, task drift, and memory poisoning, as violations of one or more security properties, thereby providing precise and contextual definitions of these attacks. Similarly, we reformalize defenses as mechanisms that strengthen oracle functions or perform security property checks. Finally, we discuss several important future research directions enabled by our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。