新基准测试300个跨模态复杂问题,逼出大模型真实搜索能力
BrowseComp-$V^3$: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents
- 设计跨文本与视觉的多跳推理题,证据藏在网页中需主动挖掘
- 顶尖模型仅36%准确率,暴露多模态信息融合关键短板
- 适合评估智能体深度搜索能力,推动真实场景应用发展
多模态大语言模型(MLLM)正发展为具备规划与工具使用能力的自主智能体,可在开放世界中执行多模态网页浏览与深度搜索。然而现有基准在任务复杂度、证据可获取性与评估粒度方面仍显不足,难以全面、可复现地评估深度搜索能力。为此,我们提出BrowseComp-$V^3$,一个包含300个精心设计的挑战性问题的新基准,覆盖多样领域。该基准强调深度、多层次及跨模态的多跳推理,关键证据在文本与视觉模态间交错分布于多个网页。所有支持证据均需公开可搜,确保公平与可复现性。除最终答案准确率外,引入专家验证的子目标驱动过程评估机制,实现对中间推理行为的细粒度分析与能力边界系统刻画。同时提出OmniSeeker框架,整合多种网络搜索与视觉感知工具。全面实验表明,即使最先进的模型在本基准上准确率也仅达36%,揭示了多模态信息整合与精细感知方面的重大瓶颈。结果凸显当前模型能力与真实世界鲁棒多模态深度搜索之间的根本差距。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs), equipped with increasingly advanced planning and tool-use capabilities, are evolving into autonomous agents capable of performing multimodal web browsing and deep search in open-world environments. However, existing benchmarks for multimodal browsing remain limited in task complexity, evidence accessibility, and evaluation granularity, hindering comprehensive and reproducible assessments of deep search capabilities. To address these limitations, we introduce BrowseComp-$V^3$, a novel benchmark consisting of 300 carefully curated and challenging questions spanning diverse domains. The benchmark emphasizes deep, multi-level, and cross-modal multi-hop reasoning, where critical evidence is interleaved across textual and visual modalities within and across web pages. All supporting evidence is strictly required to be publicly searchable, ensuring fairness and reproducibility. Beyond final-answer accuracy, we incorporate an expert-validated, subgoal-driven process evaluation mechanism that enables fine-grained analysis of intermediate reasoning behaviors and systematic characterization of capability boundaries. In addition, we propose OmniSeeker, a unified multimodal browsing agent framework integrating diverse web search and visual perception tools. Comprehensive experiments demonstrate that even state-of-the-art models achieve only 36% accuracy on our benchmark, revealing critical bottlenecks in multimodal information integration and fine-grained perception. Our results highlight a fundamental gap between current model capabilities and robust multimodal deep search in real-world settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。