研究AI代理如何使用记忆,发现早期错误会引发连锁失效。
The Compliance Trap: Diagnosing How AI Agents Consume Conflicting Memory
- 提出E-P-R框架,分析记忆在多步任务中的引入、传播与恢复过程。
- 实验显示:70%以上错误始于首个决策点,且后续难以修正。
- 强模型因更易受干扰,合规后成功率下降更严重。
记忆正成为长时序AI代理的核心组件,使代理能在浏览器、软件工具等交互环境中复用过往经验。现有研究多关注记忆的存储与检索问题,但对模型如何在多步动作轨迹中消费已检索的记忆缺乏清晰理解。本文提出轨迹级分析框架Entry--Propagation--Recovery(E-P-R),用于诊断记忆在何处首次影响动作、该影响是否持续传递,以及偏离正确路径后能否恢复。我们在WebArena和自建的控制性基准MemTrapBench上验证该框架。结果发现,主要失败往往始于记忆引入阶段:代理在首个决策点即采纳与任务冲突的记忆,即使明显错误。重复暴露使错误被放大,而偏离后的恢复能力极弱。这导致‘合规陷阱’现象:不同模型面对冲突记忆时的合规率相近,但一旦合规,其成功率达至极低水平。更强的代理因此遭受更大绝对损失,因每次合规都会抹除更多原始能力。研究建议:评估记忆增强型代理不应仅看检索质量或最终成功率,还应考察其全轨迹上的记忆消费行为。
原文摘要 · Abstract (English)
Memory is becoming a core component of long-horizon AI agents, allowing agents to reuse past experience when operating web browsers, software tools, and other interactive environments. Existing work mostly treats memory as a supply problem, asking what experience to write, how to store it, and which entry to retrieve for the next task. Yet we still lack a clear account of how models consume retrieved memory across a multi-step action trajectory. This consumption process matters because it determines not only what memories should be retrieved, but also what models and control policies are needed to use them safely. To diagnose this process, we propose Entry--Propagation--Recovery (E-P-R), a trajectory-level framework that asks where memory first changes an action, whether that change carries forward, and whether the agent can recover after leaving a correct path. We instantiate E-P-R on WebArena and on MemTrapBench, a controlled benchmark we build to isolate these phases. We find that the main failure often begins at entry: agents adopt conflicting memory at the first exposed decision point even when it is task-wrong. Repeated exposure then amplifies this early error, while recovery after divergence is weak. Together, these effects create a compliance trap: across models, conflicting memory induces similar compliance rates, but once agents comply, their success rates collapse to a low floor. Stronger agents therefore suffer larger absolute damage because each compliance event erases more baseline capability. These results suggest that memory-augmented agents should be evaluated not only by retrieval quality or final success rate, but by how they consume memory throughout the trajectory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。