评测大模型在图文混合搜索中的表现，发现GPT-4o优于现有商业产品。
MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines
- 构建MMSearch-Engine框架，让大模型具备图文检索能力。
- 在300个无训练数据重叠的实例上测试，GPT-4o在端到端任务中超越Perplexity Pro。
- 揭示当前模型仍难理解多模态搜索，提示可扩展推理计算提升性能。
大型语言模型（LLMs）推动了AI搜索引擎的发展，但多数仅支持文本查询，忽视用户多模态需求与网页中图文混排特性。尽管大模型多模态（LMMs）取得进展，其作为搜索引擎的潜力仍未被探索。为此，我们设计MMSearch-Engine管道，使任意LMM具备多模态搜索能力，并提出MMSearch基准评估其性能。该数据集包含300个手动收集的实例，覆盖14个子领域，且与现有LMM训练数据无重叠，确保答案需通过搜索获得。使用该管道，对闭源与开源LMMs进行多项任务评估：重查询、重排序、摘要生成及完整端到端搜索。实验表明，GPT-4o结合MMSearch-Engine在端到端任务中超越商业产品Perplexity Pro，验证了方案有效性。误差分析显示，当前模型仍难以完全掌握多模态搜索任务；消融研究进一步表明，增加推理时计算量有提升潜力。本工作为多模态AI搜索引擎发展提供新视角。
原文摘要 · Abstract (English)
The advent of Large Language Models (LLMs) has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine. Project Page: https://mmsearch.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。