构建真实场景下的多模态搜索评估基准,提升模型视觉检索能力。
Vision-DeepResearch Benchmark: Rethinking Visual and Textual Search for Multimodal Large Language Models
- 设计多阶段审核的2000个问答实例,确保需视觉搜索才能回答。
- 提出多轮裁剪搜索流程,显著提升模型在真实视觉检索中的表现。
- 适合研究多模态深度研究系统与评估方法的学者使用。
多模态大语言模型(MLLMs)已推动视觉问答(VQA)发展,并支持利用搜索引擎进行复杂视觉-文本事实检索的视觉深研系统。然而,评估这些视觉与文本搜索能力仍具挑战性,现有基准存在两大局限:一是非视觉搜索导向,问题答案常可通过文本线索或模型先验知识推断;二是评估场景过于理想化,图像搜索依赖近似匹配,文本搜索则过于直接。为此,我们构建了视觉深研基准(VDR-Bench),包含2,000个经多阶段严格筛选与专家评审的问答实例,旨在评估视觉深研系统在真实世界条件下的表现。针对当前MLLM视觉检索能力不足的问题,我们提出一种简单有效的多轮裁剪搜索工作流,实验证明其能显著提升模型在真实视觉检索场景下的性能。结果为未来多模态深度研究系统的构建提供实践指导。代码将发布于 https://github.com/Osilly/Vision-DeepResearch。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have advanced VQA and now support Vision-DeepResearch systems that use search engines for complex visual-textual fact-finding. However, evaluating these visual and textual search abilities is still difficult, and existing benchmarks have two major limitations. First, existing benchmarks are not visual search-centric: answers that should require visual search are often leaked through cross-textual cues in the text questions or can be inferred from the prior world knowledge in current MLLMs. Second, overly idealized evaluation scenario: On the image-search side, the required information can often be obtained via near-exact matching against the full image, while the text-search side is overly direct and insufficiently challenging. To address these issues, we construct the Vision-DeepResearch benchmark (VDR-Bench) comprising 2,000 VQA instances. All questions are created via a careful, multi-stage curation pipeline and rigorous expert review, designed to assess the behavior of Vision-DeepResearch systems under realistic real-world conditions. Moreover, to address the insufficient visual retrieval capabilities of current MLLMs, we propose a simple multi-round cropped-search workflow. This strategy is shown to effectively improve model performance in realistic visual retrieval scenarios. Overall, our results provide practical guidance for the design of future multimodal deep-research systems. The code will be released in https://github.com/Osilly/Vision-DeepResearch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。