反思型智能体可能记住错误任务理解并反复犯错。
Honest Lying: Understanding Memory Confabulation in Reflexive Agents
- 用程序化信号替代自我诊断，避免记忆错误
- 在ALFWorld中将正确目标提及率从0%提升至86%
- 适合研究智能体自我反思机制的开发者参考
反思型智能体依赖自生成的反思作为记忆，隐含假设其能准确诊断自身失败。我们发现这一假设可能系统性失效：在ALFWorld和HumanEval中，智能体存储了自信但错误的任务理解，并在每次环境重置后仍持续依此行动。这种现象称为记忆错构，我们提出基于日志的反射重复率（RRR）来检测对错误反思内容的重复依赖。利用RRR，我们在ALFWorld中识别出16个冻结环境，其中121次反思均未提及正确目标对象；在HumanEval中发现4个类似案例。通过将开放式自我诊断替换为轨迹级失败信号的程序化提取，正确目标提及率从0%提升至86%，RRR从0.64降至0.10，并成功解决3个冻结环境，表明反思记忆可能强化错误信念而非纠正错误。
原文摘要 · Abstract (English)
Reflexion-style agents rely on self-generated reflections as memory, implicitly assuming that agents can accurately diagnose their own failures. We show that this assumption can fail systematically: across ALFWorld and HumanEval, agents store confident but incorrect interpretations of the task and continue acting on them across trials, even though the environment resets to the correct task each time. We call this failure mode memory confabulation and introduce the Reflection Repetition Rate (RRR), a log-based metric that detects repeated reliance on incorrect reflective content. Using RRR, we identify 16 frozen environments in ALFWorld, where 0 of 121 reflections mention the correct target object, and 4 analogous cases in HumanEval. Our mitigation replaces open-ended self-diagnosis with programmatic extraction of trajectory-level failure signals, increasing correct object mention from 0% to 86%, reducing RRR from 0.64 to 0.10, and solving 3 of 16 frozen ALFWorld environments, suggesting that reflective memory can reinforce false beliefs rather than correct them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。