大模型记忆会把模糊说法变成确信事实，导致错误执行。
Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts
- 将对话中模糊表述转为确定性记忆，制造虚假可信度
- 即使无攻击者，只要一次未修正的陈述就可被永久信任
- 关键在记忆中的语气而非来源，冗余信息能防止误判
LLM代理通过压缩记忆在步骤和会话间传递结论，记忆产品（如mem0、LangMem）将对话重写为存储的‘事实’，后续步骤会信赖这些内容。我们发现这种重写会人为制造信心：在构建的代理环境中，一句随意、含糊的表述会变为带有日期的确定性断言，代理随后像验证过的事实一样执行，无条件批准所有高于权限的请求。无需攻击者介入：只要某角色曾真实存在且从未被纠正，就会被当作固定事实存储并执行。我们进一步发现，代理响应的不是信息来源——无论是有归属、无归属，还是伪造的‘系统记录’声明，效果相同。真正影响行为的是表述的语气：含糊词被忽略，确定性断言则被服从，且不依赖特定关键词。但并非所有含糊表达都等价：证据性语态（如‘据报道’）最不易被低估，多数模型将其视为确定断言。常见修复方法无效：被动‘未验证’标签被忽略，主动‘不要信任’指令反而强化正确记忆，只能通过拒绝决策来保证安全。真正的解决之道在于存储环节：保持原始的试探性表述，而非升级为确定结论。但这只是卫生习惯，并不能防御攻击者直接写下自信谎言。可部署的教训更具体而积极：单一关键记忆是风险源，引入冗余信息源可恢复正确判断。我们已发布工具包与演示。
原文摘要 · Abstract (English)
LLM agents carry conclusions across steps and sessions in compressed memory, and memory products (e.g., mem0, LangMem) rewrite conversation into stored "facts" that later steps trust. We show this rewriting manufactures confidence: across our constructed agent settings, a casual, hedged remark becomes a confident, dated assertion the agent then obeys like a verified fact, granting every above-clearance request it faces. No attacker is needed: a role that was true once and never corrected is stored as a flat fact and acted on like a deliberate injection. We then isolate what the agent responds to. It is not the source: attributed, unattributed, and even forged "system of record" claims all grant alike. It is the confidence of the phrasing. A hedge is discounted, a flat assertion is obeyed, and this holds with no special keyword. Not all hedges are equal, though: the evidential register is the least-discounted, with "reportedly" obeyed like a flat assertion on most models. The obvious fixes fail. A passive "unverified" tag is ignored, and an active "do not trust this" instruction escalates even correct memory, so it is safe only by refusing to decide. The real fix lives in the store: keep the tentative phrasing rather than upgrade it. But that is hygiene, not a defense against an attacker who can simply write a confident lie. The deployable lesson is narrower and constructive: a single load-bearing memory is the hazard, and one redundant source restores correct decisions. We release the harness and demonstrations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。