首个测试多模态智能体视觉记忆的交互式基准,揭示其遗忘机制。
DMV-Bench: Diagnosing Long-Horizon Multimodal Agents' Visual Memory with Incidental Cue Injection

- 构建可控家居电商环境,注入隐性视觉线索检验记忆
- 新架构DualMem在多轮任务中显著优于基线,提升记忆保留率
- 双编码机制可分工协作,适合需长期记忆的AI系统研究
现有智能体记忆评估多聚焦文本,仅考察主动记录内容,忽略实际感知信息。本文提出首个交互式多模态视觉记忆基准DMV-Bench,基于包含1000种商品变体的家居电商环境,通过文本泄漏约束确保任务判别信号仅来自像素。智能体在自主购物会话中访问带唯一预渲染隐性线索的产品图像,后续需回忆。传统方法表现不佳。受双重编码理论启发,提出双编码记忆架构DualMem,使用并行视觉与语言代码。在多种模型和多会话链长下,DualMem超越仅用描述词的基线及三种近期多模态记忆系统,性能优势在调整记忆库大小和编码位置偏差后仍存。进一步实验发现两种编码存在非对称分工,加权编码方案通常最优。该工作推动更完整记录智能体观测的内存系统发展。
原文摘要 · Abstract (English)
Agent benchmarks for measuring memory largely study textual cases, in which information is deliberately extracted from the environment, written down, and then later retrieved. In other words, they assess what agents elected to record, not what they happened to see. We introduce DMV-Bench (code: https://github.com/yyyujintang/DMV-Bench), the first interactive benchmark for visual memory in multimodal agents, to study this often-neglected property. DMV-Bench is built on (1) a controlled home-furnishing e-commerce environment, supported by a catalog of 1,000 product variants, and (2) a text-leakage contract which ensures that the primary discriminative signal of each task is solely in the pixels. In DMV-Bench, agents undergo chains of autonomous shopping sessions in which every visited product image carries a unique, pre-rendered incidental cue that the agent is later asked to recall. We show that conventional solutions struggle with this task. Inspired by dual-coding theory, we propose a memory architecture that uses parallel visual and verbal codes, which we call DualMem. On DMV-Bench, DualMem outperforms a caption-only baseline and three recent multimodal agent-memory systems across multi-session chain lengths on multiple models. These gains persist even adjusting for memory-bank size and encoding-position bias. Further experiments also reveal an asymmetric division of labor between the two codes; a weighted coding scheme is often strongest. We view this as a step towards memory systems that preserve a richer record of agents' observations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。