探究多模态大模型在图像类比推理中的短板，发现其通用推理能力不足。
Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems
- 设计多种适合多模态模型的解题策略，测试8个模型在3个数据集上的表现。
- 在真实世界图像上表现尚可，但在经典合成图像类比题上普遍失败。
- 提出新数据集Bongard-RWR，证明问题根源是通用推理缺陷而非领域差异。
抽象视觉推理（AVR）涉及通过类比发现图像间的共享概念，类似于解决智商测试题。博加德问题（BPs）仍是AVR的核心挑战，要求同时具备视觉推理与语言描述能力。本文研究多模态大语言模型（MLLMs）是否能解决BPs，设计了一组适配MLLM的解题策略，并在包含合成图像（经典BPs）和真实世界图像（Bongard HOI、Bongard-OpenWorld）的3个数据集上测试了4个专有模型和4个开源模型。尽管在真实世界数据集上取得一定成功，但MLLMs在合成图像任务中表现不佳。为探究这一差距，我们引入了Bongard-RWR数据集，用真实世界图像表示经典合成问题的概念。结果表明，MLLM在经典BPs上表现差并非源于领域特异性，而是其通用视觉推理能力的局限。代码与数据集见：https://github.com/pavonism/bongard-rwr
原文摘要 · Abstract (English)
Abstract visual reasoning (AVR) involves discovering shared concepts across images through analogy, akin to solving IQ test problems. Bongard Problems (BPs) remain a key challenge in AVR, requiring both visual reasoning and verbal description. We investigate whether multimodal large language models (MLLMs) can solve BPs by formulating a set of diverse MLLM-suited solution strategies and testing $4$ proprietary and $4$ open-access models on $3$ BP datasets featuring synthetic (classic BPs) and real-world (Bongard HOI and Bongard-OpenWorld) images. Despite some successes on real-world datasets, MLLMs struggle with synthetic BPs. To explore this gap, we introduce Bongard-RWR, a dataset representing synthetic BP concepts using real-world images. Our findings suggest that weak MLLM performance on classical BPs is not due to the domain specificity, but rather comes from their general AVR limitations. Code and dataset are available at: https://github.com/pavonism/bongard-rwr
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。