arXiv:2509.26378cs.IRcs.CV2025-09被引 6

构建首个需深度推理的多模态检索评测基准,挑战模型逻辑与因果理解能力。

MR$^2$-Bench: Going Beyond Matching to Reasoning in Multimodal Retrieval

  • 所有任务均需逻辑、空间、因果推理,超越表面语义匹配。
  • 涵盖1309个精心设计查询,覆盖图像、图表、视觉谜题等多样内容。
  • 适合研究多模态推理、智能检索系统的学者与开发者使用。

多模态检索已成为现代AI应用的关键组件,但其评估仍难以满足更真实、更具挑战性场景的需求。现有基准主要考察表层语义对应(如物体-文本匹配),却无法评估视觉与文本信息间复杂关系所需的深层推理能力。为此,我们提出MR$^2$-Bench,一个以推理为核心的多模态检索评测基准。该基准具备三大核心价值:1)所有任务均为推理驱动,突破浅层匹配,有效检验模型在逻辑、空间和因果推理方面的能力;2)包含自然图像、图表、视觉谜题等多样化多模态数据,实现跨内容类型的全面评估;3)支持包含多张图片的复杂查询与文档,覆盖多样检索场景,更贴近真实应用。基准共含1,309个经过筛选与标注的查询,部分来自人工收集,部分源自公开数据集的有选择性整合。尽管当前领先模型在已有基准上表现优异,但在MR$^2$-Bench上仍面临巨大挑战:例如,领先模型Seed1.6-Embedding在MMEB上的Recall@1为77.78,而在本基准上仅为9.91。这一显著性能差距凸显了本基准的难度提升,也反映出多模态推理能力亟待突破。数据集与评估代码将公开于https://github.com/VectorSpaceLab/MR2-Bench。

原文摘要 · Abstract (English)

Multimodal retrieval is becoming a crucial component of modern AI applications, yet its evaluation lags behind the demands of more realistic and challenging scenarios. Existing benchmarks primarily probe surface-level semantic correspondence (e.g., object-text matching) while failing to assess the deeper reasoning required to capture complex relationships between visual and textual information. To address this gap, we introduce MR$^2$-Bench, a reasoning-intensive benchmark for multimodal retrieval. MR$^2$-Bench presents the following critical values: 1) all tasks are reasoning-driven, going beyond shallow matching to effectively assess models' capacity for logical, spatial, and causal inference; 2) it features diverse multimodal data, such as natural images, diagrams, and visual puzzles, enabling comprehensive evaluation across content types; 3) it supports complex queries and documents containing multiple images and covers diverse retrieval scenarios, more accurately reflecting real-world applications. Our benchmark contains 1,309 curated queries, derived either from manual collection and annotation or from selective consolidation of public datasets. Despite achieving strong results on existing benchmarks, current state-of-the-art models still struggle on MR$^2$-Bench: for example, the leading Seed1.6-Embedding model attains a Recall@1 of 77.78 on MMEB, but only 9.91 on MR$^2$-Bench. This substantial performance gap highlights both the increased challenge posed by our benchmark and the pressing need for further advances in reasoning-intensive multimodal retrieval. The dataset and evaluation code will be made publicly available at https://github.com/VectorSpaceLab/MR2-Bench.

多模态检索推理能力评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。