评测大模型在多轮跨模态对话中的长期记忆能力，发现现有方法各有短板。
MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
- 构建跨会话多模态记忆评测集，涵盖五类能力与四种上下文长度。
- 视觉信息缺失导致顶尖模型准确率降至2%以下，证明需多模态证据。
- 长上下文模型易退化，记忆代理失真，二者需融合提升性能。
记忆对大型视觉语言模型（LVLMs）处理长时多模态交互至关重要，当前主要依赖长上下文架构和记忆增强型智能体两类方法。然而，现有评估基准缺乏对两类方法在真实需要多模态证据任务上的系统性比较。为此，我们提出 MEMLENS，一个全面的多轮跨会话多模态记忆评测基准，包含789个问题，覆盖五种记忆能力（信息提取、多会话推理、时间推理、知识更新、答案拒绝），在四个标准上下文长度（32K–256K tokens）下进行测试，采用跨模态分词计数方案。图像消融实验表明：移除视觉证据使两个前沿LVLM在含图像证据的80.4%问题上准确率低于2%。评估27个LVLM和7个记忆增强智能体发现，长上下文模型在短上下文下表现优异但随对话增长显著退化；记忆智能体具备长度稳定性，但在存储压缩中损失视觉保真度。多会话推理普遍低于30%，单一方法无法解决该任务。结果表明需结合长上下文注意力与结构化多模态检索的混合架构。代码已开源于 https://github.com/xrenaf/MEMLENS。
原文摘要 · Abstract (English)
Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K-256K tokens) under a cross-modal token-counting scheme. An image-ablation study confirms that solving MEMLENS requires visual evidence: removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images. Evaluating 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow, whereas memory agents are length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval. Our code is available at https://github.com/xrenaf/MEMLENS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。