构建真实用户-代理交互的多模态记忆基准，评测模型对隐含信息的理解能力。
M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
- 基于真实交互设计查询中心的多模态对话记忆评测框架
- 发现多模态大模型在跨模态对齐与会话推理上仍有显著差距
- 提出按需调用视觉源的方法，准确率提升13%，效率提升超70%
语言代理日益依赖累积的多模态信息，但现有基准仍沿用人类间交互范式，仅含稀疏视觉和简单内容，无法评估真实场景下的多模态文件交互推理或隐藏用户信息的解读。为此，我们提出 M$^3$Exam，一个基于真实用户-代理交互的查询中心型多模态对话记忆基准，涵盖跨模态对齐与隐含信息推断的多维度评估。在 MLLMs 与记忆系统上的基准测试揭示了跨模态对齐、跨会话推理及多模态上下文累积带来的效率代价等持续存在的差距。我们进一步提出 M$^3$Proctor，一种能检测查询模态偏差、仅在需要时调用原始视觉源的多模态记忆方法，在准确率提升 13% 的同时，将索引构建时间与检索标记数减少超过 70%。
原文摘要 · Abstract (English)
Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over authentic multimodal file interaction nor the interpretation of concealed user information. We therefore introduce M$^3$Exam, a query-centric multimodal conversational memory benchmark built on realistic user-agent interaction, with multi-dimensional evaluation spanning cross-modal grounding and implicit information inference. Benchmarking MLLMs and memory systems reveals persistent gaps in cross-modal grounding, cross session reasoning, and the efficiency cost of accumulating multimodal context. We further propose M$^3$Proctor, a multimodal memory method that detects query modality bias and consumes raw visual sources only on demand, improving accuracy by 13% while cutting index-construction time and retrieved tokens by over 70%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。