构建首个多模态代理记忆评估框架,可诊断记忆各阶段表现
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction

- 提出记忆四阶段循环模型,基于真实交互任务评估
- 400个任务实测显示视觉信息利用不足且系统跨域不稳定
- 支持手动与自管理记忆系统的直接对比,适合长时序智能体研究者
多模态大语言模型正越来越多地作为长周期智能体部署,此时记忆不仅要能回忆,还需追踪动态世界、更新过时内容,并在决策时准确调用证据。现有基准仅评估静态对话中的记忆召回,将记忆性能简化为单一任务结束精度,且将视觉观察降维为文本描述,无法定位失败发生在写作、维护、检索还是使用阶段。随着代理自主生成记忆的兴起,这一差距愈发显著,因缺乏可比性方法来评估人工设计管道与自管理方案的优劣。为此,本文将多模态代理记忆建模为可观测的“行动-世界交互循环”,包含四个阶段生命周期,并构建了WorldMemArena:涵盖400个多会话多模态任务,覆盖长期演化(个人与任务状态演变)与代理执行(基于真实观察、动作与反馈的记忆),并标注了黄金记忆点、更新记录、干扰项与证据链,支持阶段级诊断。该框架首次实现对长上下文、人工设计(RAG与外部记忆系统)与基于钩子(harness)的记忆代理的直接比较。结果表明:(1) 更好的记忆写入与存储不等于更高性能;(2) 多模态记忆仍难以充分使用视觉证据;(3) 系统在不同领域间表现不稳定,在真实代理轨迹上性能下降;(4) 钩子式记忆更灵活但成本高且可靠性低。
原文摘要 · Abstract (English)
Multimodal large language models are increasingly deployed as long-horizon agents, where memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance, retrieval, or use. The rise of agent harnesses that author their own memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate multimodal agent memory as an Action-World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session multimodal tasks spanning Lifelong Evolution (evolving personal and task states) and Agentic Execution (memory from real observations, actions, and feedback), annotated with gold memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external memory systems), and harness-based memory agents. Results show that: (1) better memory writing and storage do not guarantee better performance; (2) multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness memory is more flexible but remains costly and less reliable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。