首个面向多模态复杂推理的检索基准，挑战现有模型视觉理解能力。
MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval
- 构建跨29个技术领域的真实多模态查询数据集，涵盖四类递增难度任务。
- 顶尖模型在图文检索任务中仅达27.6 nDCG@10，低于最佳纯文本模型。
- 适合研究多模态推理、信息检索与视觉语言模型的学者使用。
现有检索基准主要基于文本查询，关键词或语义匹配通常已足够。但许多真实场景中的查询包含图像等多模态元素，如图表、截图，需深入推理才能定位相关文档。为此，我们提出首个面向复杂推理的多模态检索基准MM-BRIGHT。数据集包含2,803个真实世界查询，覆盖29个不同技术领域，设有四类逐步增加复杂度的任务：文本到文本、多模态到文本、多模态到图像、多模态到多模态检索。大量评估显示，当前最优模型在所有任务上表现不佳：BM25在纯文本检索中仅获8.5 nDCG@10；最佳多模态模型Nomic-Vision在多模态到文本检索中仅达27.6 nDCG@10，甚至低于最佳纯文本模型DiVeR的32.2。这些结果揭示了显著的提升空间，凸显了MM-BRIGHT作为下一代融合视觉推理检索模型测试平台的价值。代码与数据已开源于https://github.com/mm-bright/MM-BRIGHT，官网为https://mm-bright.github.io/。
原文摘要 · Abstract (English)
Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。