新基准测试让视觉语言模型暴露真实短板
Beyond Visual Understanding: Introducing PARROT-360V for Vision Language Model Benchmarking
- 设计2487道复杂视觉谜题，考验多模态推理能力
- 顶尖模型在新基准上得分仅28%至56%
- 适合研究多模态推理与评估框架的学者
当前视觉语言模型（VLMs）的评估基准往往难以全面检验模型对复杂视觉与文本内容的理解和处理能力。它们通常聚焦于简单任务，缺乏深度推理或跨模态融合来解决原创问题。为此，我们提出了PARROT-360V基准，一个包含2487个高挑战性视觉谜题的新基准，旨在测试VLM在复杂视觉推理任务中的表现。我们使用该基准评估了GPT-4o、Claude-3.5-Sonnet和Gemini-1.5-Pro等领先模型，考察其结合视觉线索与语言能力解决任务的能力，类似人类问题解决方式。结果显示显著性能差距：最先进模型在该基准上的得分仅为28%至56%，远低于其在主流基准上的表现。这凸显了现有VLM在处理复杂多步推理任务时的局限性，并强调了构建更强大评估框架以推动领域发展的必要性。
原文摘要 · Abstract (English)
Current benchmarks for evaluating Vision Language Models (VLMs) often fall short in thoroughly assessing model abilities to understand and process complex visual and textual content. They typically focus on simple tasks that do not require deep reasoning or the integration of multiple data modalities to solve an original problem. To address this gap, we introduce the PARROT-360V Benchmark, a novel and comprehensive benchmark featuring 2487 challenging visual puzzles designed to test VLMs on complex visual reasoning tasks. We evaluated leading models: GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro, using PARROT-360V to assess their capabilities in combining visual clues with language skills to solve tasks in a manner akin to human problem-solving. Our findings reveal a notable performance gap: state-of-the-art models scored between 28 to 56 percentage on our benchmark, significantly lower than their performance on popular benchmarks. This underscores the limitations of current VLMs in handling complex, multi-step reasoning tasks and highlights the need for more robust evaluation frameworks to advance the field.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。