测试大模型何时该记住、验证或询问用户，避免错误记忆影响行为。
Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agents
- 设计140个场景评估模型对信息是否持久化、验证或询问的决策能力。
- 少样本提示使准确率从55.7%提升至77.1%，但澄清召回仍仅33.3%。
- 模型更倾向自验证而非问用户，适合研究记忆安全与人机交互的开发者。
持久记忆可个性化大模型智能体，但错误的长期更新可能悄然扭曲后续行为。本文研究记忆确认边界：交互获取的信息应持久保存、仅在当前上下文使用、重新验证，还是向用户澄清。MCB包含140个主场景（70个开发集+70个保留集）及70个对比集，评估动作标签与结构化工具调用选择。两名非作者独立标注保留集和对比集（97.1%一致，Cohen's kappa=0.962），盲评第三方解决4处分歧，替换8个作者标签。在Claude与Qwen模型中，模型更可靠地验证事实变更，而非请求用户澄清歧义。裸版Qwen在12个澄清项中未提问（0/12），却正确验证了18项中的12项（12/18）。少样本提示将准确率从0.557提升至0.771（配对差值+0.214，霍尔姆校正精确麦内马尔p_H=0.002），但澄清召回率仅为0.333。策略提示将错误持久化从0.243降至0.100（p_H=0.038），尽管准确率提升不显著。工具调用与标签一致性为：每款Claude模型57%，Qwen仅23%；Qwen准确率从0.557降至0.343（p_H=0.047）。记忆评估需同时检验决策声明与工具调用选择。
原文摘要 · Abstract (English)
Persistent memory can personalize an LLM agent, but an incorrect durable update can silently distort future behavior. We study the memory-clarification boundary: whether interaction-derived information should be persisted, used only in the current context, re-verified, or clarified with the user. MCB contains 140 primary scenarios, split into 70 development and 70 held-out items, plus a separate 70-item contrast set. It evaluates both action labels and structured tool-call selection. Two non-authors independently label the 70 held-out primary and 70 contrast items (97.1% agreement, Cohen's kappa = 0.962); a blind third resolves four disagreements, replacing eight author labels by non-author majority. Across Claude and Qwen, models verify changing facts more reliably than they ask users to resolve ambiguity. Bare Qwen asks on 0/12 clarification items while verifying 12/18 freshness items. Few-shot prompting raises accuracy from 0.557 to 0.771 (paired delta = +0.214, Holm-adjusted exact McNemar p_H = 0.002), yet clarification recall remains 0.333. The policy prompt reduces erroneous persistence from 0.243 to 0.100 (p_H = 0.038), although its accuracy gain is not significant. Label-tool agreement is 57% for each Claude model and 23% for Qwen; Qwen accuracy falls from 0.557 to 0.343 (p_H = 0.047). Memory evaluation must test both stated decisions and tool-call choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。