黑客利用大模型记忆系统窃取敏感信息,攻击成功率高达100%。
Trojan Hippo: Weaponizing Agent Memory for Data Exfiltration

- 通过一次恶意工具调用植入潜伏记忆,仅在用户谈敏感话题时激活
- 在四种记忆架构上测试,攻击成功率达85%-100%,百次正常会话后仍有效
- 首次系统评估防御措施的安全部署代价,揭示安全与功能的权衡难题
记忆系统使大模型代理能在会话间持久保存用户信息,但也引入新攻击面。本文提出Trojan Hippo攻击,一种更贴近现实威胁模型的持久记忆攻击:攻击者通过一次不可信工具调用(如伪造邮件)将潜伏载荷植入代理长期记忆,仅当用户后续讨论金融、健康或身份等敏感话题时才激活,并向攻击者外泄高价值个人数据。尽管此类攻击已在部署系统中出现过案例,但此前缺乏对异构记忆架构与防御机制的系统性评估。为此,我们构建了一个动态评估框架,包含:(1) 基于OpenEvolve的自适应红队测试基准,持续迭代攻击以压力测试防御与记忆后端;(2) 首个能力感知的安全/效用分析,支持根据不同使用场景合理部署防御策略。在四类记忆架构(显式工具记忆、代理记忆、RAG、滑动窗口上下文)上的电子邮件助手实验表明,该攻击对OpenAI和Google的前沿模型可达85%-100%攻击成功率,且在经历100次良性会话后仍可激活。我们评估了四种基于基础安全原则的记忆系统防御方案,发现其可将攻击成功率降至0%-5%,但伴随显著的效用损失,且损失程度随任务需求差异巨大。因此,防御的实际部署面临重大挑战,而本框架正是为此设计。
原文摘要 · Abstract (English)
Memory systems enable otherwise-stateless LLM agents to persist user information across sessions, but also introduce a new attack surface. We characterize the Trojan Hippo attack, a class of persistent memory attacks that operates in a more realistic threat model than prior memory poisoning work: the attacker plants a dormant payload into an agent's long-term memory via a single untrusted tool call (e.g., a crafted email), which activates only when the user later discusses sensitive topics such as finance, health, or identity, and exfiltrates high-value personal data to the attacker. While anecdotal demonstrations of such attacks have appeared against deployed systems, no prior work systematically evaluates them across heterogeneous memory architectures and defenses. We introduce a dynamic evaluation framework comprising two components: (1) an OpenEvolve-based adaptive red-teaming benchmark that stress-tests defenses and memory backends against continuously refined attacks, and (2) the first capability-aware security/utility analysis for persistent memory systems, enabling principled reasoning about defense deployment across different usage profiles. Instantiated on an email assistant across four memory backends (explicit tool memory, agentic memory, RAG, and sliding-window context), Trojan Hippo achieves up to 85-100% ASR against current frontier models from OpenAI and Google, with planted memories successfully activating even after 100 benign sessions. We evaluate four memory-system defenses inspired by basic security principles, finding they substantially reduce attack success rates (to as low as 0-5%), though at utility costs that vary widely with task requirements. Because of this substantial security-utility tradeoff, the effective real-world deployment of defenses remains an open challenge, which our evaluation framework is specifically designed to address.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。