arXiv:2605.22842cs.CRcs.AI2026-05

记忆污染伪装成模型失效,导致安全防护误判。

The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems

论文配图:The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems
图 1 · 摘自论文原文
  • 提出语义规范漂移机制,解释记忆攻击如何伪装成模型问题
  • 97%攻击在跨会话边界被新防御方案拦截,510个检查点零漏检
  • 适合安全研究者、多智能体系统开发者关注

多智能体系统通常认为代理行为异常源于模型对齐失败。本文发现这一假设存在结构性漏洞——‘误归因缺口’,即记忆层攻击产生的行为与模型失效完全无法区分,导致防御方采取错误修复措施。我们形式化了‘语义规范漂移’(SND)作为第三类代理违规路径,不同于涌现对齐偏差和合谋。一个策略性文档通过正常上传进入共享向量存储,经由信任清洗链后失去来源痕迹,再次出现时被当作可信系统上下文。在64起已记录故障中,归因系统始终归咎于模型。四个安全分类器(含一个训练于记忆污染数据的)在510个检查点上均未检测到任何攻击。在65个有效案例中,59例代理在执行前明确引用该注入文档作为规范依据。该攻击无需触发条件、模型访问或重复交互,五次会话内完成且永久持续。我们提出反事实组合测试,在25个场景中以87.5%准确率实现零误报,而传统取证基线全数失败。进一步证明‘检索覆盖困境’,表明更强的规避能力反而削弱攻击,限制自适应绕过策略。最后提出内存持久型信息流控制,在跨会话边界阻断97%攻击,突破此前防御盲区。发布SND语料库,首个具备时间持久性与多智能体组合特性的对抗记忆基准,涵盖金融与医疗领域。

原文摘要 · Abstract (English)

Multi-agent AI pipelines typically assume that agent misconduct originates from model misalignment. We identify a structural failure in this assumption, the \emph{Misattribution Gap}, where memory-layer attacks produce behaviors indistinguishable from model failure, causing defenders to apply the wrong remediation. We formalize \emph{Semantic Norm Drift} (SND) as a third path to agent misconduct, distinct from emergent misalignment and collusion. In SND, a policy-formatted document enters a shared vector store through normal uploads and later reappears as trusted system context after provenance is lost through a Trust Laundering Chain. Across 64 documented failures, attribution systems consistently blamed the model. Four safety classifiers, including one trained on memory poisoning, produced zero detections across 510 checkpoints. In 59 of 65 valid cases, agents explicitly cited the injected document as normative authority before complying. The attack requires no trigger, model access, or repeated interaction, achieves full effect within five sessions, and persists indefinitely. We introduce Counterfactual Composition Testing, which identifies the causal entry with 87.5% accuracy and zero false positives, while a forensics baseline fails across all 25 scenarios. We further prove the Retrieval-Coverage Dilemma, showing that stronger evasion inherently weakens the attack, limiting adaptive bypass strategies. Finally, we propose Memory-Persistent Information-Flow Control, which blocks 97% of attacks at the cross-session boundary where prior defenses fail. We release the SND Corpus, the first adversarial memory benchmark with temporal persistence and multi-agent composition across financial and Health Care domains.

记忆攻击多智能体安全防御语义漂移

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。