防止大模型记忆被伪造，确保动作来源可信
Memory Provenance Laundering in LLM Agents: A Non-Amplification Firewall for Persistent Memory
- 设计轻量级中间件保留记忆来源证据，防止恶意信息伪装成用户历史
- 实验显示未防护时攻击成功率高达1.000，防护后零越权高风险操作
- 适合需要安全记忆管理的AI代理系统开发者使用
长期记忆使大语言模型代理能够复用先前偏好和工作流，但也可能将不可信的观测转化为持久的动作上下文。我们识别出‘记忆溯源清洗’问题：在基于LLM的记忆整合过程中，外部观察可能被重写为看似用户历史或流程支持，保留动作触发但抹去应限制其权限的低信任来源。现有提示过滤、内容净化和工具防护机制无法在丢失性记忆整合后保证源权威性不被放大。我们形式化该边界，并实现为溯源保持型内存防火墙（PPMF），一种轻量级内存中间件，通过保留平台维护的溯源信息、确认状态和风险标签，匹配动作风险与相关记忆的权威性来授权工具调用。在固定风险策略的场景化评估中，存在漏洞的整合记忆攻击成功率最高达1.000；而当平台维护的溯源、确认和风险标签完整时，所有未经批准的高风险动作均被PPMF拦截，同时已确认的良性动作及目标低风险记忆使用仍可执行。
原文摘要 · Abstract (English)
Long-term memory lets large language model(LLM) agents reuse prior preferences and work flows, but it also turns untrusted observations into persistent action context. We identify memory provenance laundering: during LLM-based memory consolidation, an external observation may be rewritten as apparent user history or workflow support, preserving an action trigger while erasing the low-trust source that should limit its authority. Existing prompt filters, content sanitizers, and tool guards do not enforce source-authority non-amplification after lossy memory consolidation. We formalize this boundary and instantiate it as Provenance-Preserving Memory Fire wall (PPMF), a lightweight memory middleware that preserves platform-maintained provenance and authorizes tool calls by matching action risk to the authority of action-relevant memories. In our schema-grounded evaluation with fixed risk policies, vulnerable consolidated memories reach up to 1.000 attack success rate(ASR); with intact platform-maintained provenance, confirmation, and risk labels, no evaluated unauthorized high-risk action passes the PPMF gate while confirmed benign actions and targeted low-risk memory use remain executable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。