利用聊天模板伪装攻击,让大模型误执行恶意指令
ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents
- 用对话模板格式包装恶意指令,诱导模型执行
- 多轮对话攻击成功率最高达52.33%,远超传统方法
- 现有防御失效,尤其对闭源模型仍有效
基于大语言模型(LLM)的智能体在与外部环境交互时面临新型攻击面。间接提示注入攻击通过在外部输出中嵌入恶意指令,使智能体将其误认为合法提示并执行。现有研究多聚焦于纯文本攻击,我们发现一个被忽视的漏洞:LLM对结构化聊天模板的依赖及其易受多轮对话中的说服性内容影响。为此,我们提出ChatInject攻击,将恶意载荷格式化为符合原生聊天模板的形式,利用模型的指令遵循倾向实施攻击。进一步构建了基于说服力的多轮变体,在多轮对话中逐步诱导智能体接受可疑行为。在前沿LLM上的实验表明:(1) ChatInject平均成功率显著提升,AgentDojo上从5.18%升至32.05%,InjecAgent上从15.13%升至45.90%,多轮对话平均成功率达52.33%;(2) 基于聊天模板的攻击具有强迁移性,即使在闭源模型上仍有效,尽管其模板结构未知;(3) 现有基于提示的防御对本攻击基本无效,尤其对多轮变体效果差。这些结果揭示了当前智能体系统的严重安全漏洞。
原文摘要 · Abstract (English)
The growing deployment of large language model (LLM) based agents that interact with external environments has created new attack surfaces for adversarial manipulation. One major threat is indirect prompt injection, where attackers embed malicious instructions in external environment output, causing agents to interpret and execute them as if they were legitimate prompts. While previous research has focused primarily on plain-text injection attacks, we find a significant yet underexplored vulnerability: LLMs' dependence on structured chat templates and their susceptibility to contextual manipulation through persuasive multi-turn dialogues. To this end, we introduce ChatInject, an attack that formats malicious payloads to mimic native chat templates, thereby exploiting the model's inherent instruction-following tendencies. Building on this foundation, we develop a persuasion-driven Multi-turn variant that primes the agent across conversational turns to accept and execute otherwise suspicious actions. Through comprehensive experiments across frontier LLMs, we demonstrate three critical findings: (1) ChatInject achieves significantly higher average attack success rates than traditional prompt injection methods, improving from 5.18% to 32.05% on AgentDojo and from 15.13% to 45.90% on InjecAgent, with multi-turn dialogues showing particularly strong performance at average 52.33% success rate on InjecAgent, (2) chat-template-based payloads demonstrate strong transferability across models and remain effective even against closed-source LLMs, despite their unknown template structures, and (3) existing prompt-based defenses are largely ineffective against this attack approach, especially against Multi-turn variants. These findings highlight vulnerabilities in current agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。