提出视觉房间2.0,测试多模态大模型‘看见’但不‘理解’的问题
Visual Room 2.0: Seeing is Not Understanding for MLLMs
- 构建三级评估框架,从感知到认知逐步深入
- 10个主流模型中认知能力普遍弱于感知能力,差距达8.0%
- 发现认知能力随模型增大而提升,但感知能力不随规模增长
多模态大语言模型(MLLMs)真的能理解所见内容吗?本文将塞尔的中文房间思想拓展至多模态领域,提出‘视觉房间’论点:MLLMs 可精准描述视觉细节,却无法理解背后的情感与意图,即‘看见≠理解’。基于此,我们构建了视觉房间2.0,一个分层级的评估基准,用于衡量 MLLMs 的感知-认知对齐程度。该框架模拟人类在低、中、高三个认知层次的感知与认知过程,涵盖17项代表性任务。感知部分包括属性识别到场景理解,认知部分涵盖文本蕴含到因果与社会推理。数据集包含350个多模态样本,每个样本有6个递进式问题(共2100个),覆盖从感知到认知的完整链条。评估10个前沿的MLLMs后,得出三个关键发现:(1) 模型感知能力显著强于认知能力(提升8.0%);(2) 认知能力并非依赖于基于感知的推理;(3) 认知能力随模型规模增长,但感知能力并不随模型变大而持续提升。本研究将‘看见≠理解’转化为可验证假设,为MLLMs从感知处理到认知推理提供新范式。数据集已公开于 https://huggingface.co/datasets/LHK2003/PCBench。
原文摘要 · Abstract (English)
Can multi-modal large language models (MLLMs) truly understand what they can see? Extending Searle's Chinese Room into the multi-modal domain, this paper proposes the Visual Room argument: MLLMs may describe every visual detail precisely yet fail to comprehend the underlying emotions and intentions, namely seeing is not understanding. Building on this, we introduce \textit{Visual Room} 2.0, a hierarchical benchmark for evaluating perception-cognition alignment of MLLMs. We model human perceptive and cognitive processes across three levels: low, middle, and high, covering 17 representative tasks. The perception component ranges from attribute recognition to scene understanding, while the cognition component extends from textual entailment to causal and social reasoning. The dataset contains 350 multi-modal samples, each with six progressive questions (2,100 in total) spanning perception to cognition. Evaluating 10 state-of-the-art (SoTA) MLLMs, we highlight three key findings: (1) MLLMs exhibit stronger perceptual competence than cognitive ability (8.0\%$\uparrow$); (2) cognition appears not causally dependent on perception-based reasoning; and (3) cognition scales with model size, but perception does not consistently improve with larger variants. This work operationalizes Seeing $\ne$ Understanding as a testable hypothesis, offering a new paradigm from perceptual processing to cognitive reasoning in MLLMs. Our dataset is available at https://huggingface.co/datasets/LHK2003/PCBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。